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PREFACE 


Since  the  late  1970s,  the  Logistics  Research  Division  of  the  Armstrong  Laboratory  has 
conducted  research  to  develop  and  evaluate  the  technology  for  an  Integrated  Maintenance 
Information  System  (IMIS).  An  IMIS  will  provide  maintenance  technicians  and  managers  with 
the  capability  to  access  all  technical  information  required  to  perform  their  jobs  via  a  single, 
integrated  system.  The  final  phase  of  the  IMIS  program  was  to  develop  a  demonstration  system 
incorporating  the  technologies  and  illustrating  the  capabilities  of  an  IMIS.  This  report 
summarizes  the  results  of  a  user  field  test  and  demonstration  conducted  to  evaluate  the  IMIS  and 
demonstrate  its  capabilities  to  operational  maintenance  personnel.  The  user  field  test  and 
demonstration  was  conducted  at  Luke  AFB,  AZ,  during  the  summer  of  1994. 

The  IMIS  is  the  product  of  the  creativity,  talents,  and  efforts  of  many  people,  including 
Armstrong  Laboratory  personnel  and  a  number  of  contractors.  The  IMIS  program  was  initiated 
and  directed  by  Mr.  Robert  C.  Johnson,  Operational  Logistics  Branch,  Armstrong  Laboratory’s 
Logistic  Research  Division  (AL/HRGO).  The  program  was  accomplished  by  a  dedicated  staff  of 
scientists  and  engineers,  with  the  strong  support  of  three  Division  Chiefs:  Mr.  Bertram  W. 
Cream,  Col.  James  C.  Clark,  and  Col.  Donald  C.  Tetmeyer. 

The  prime  contractor  for  the  development  of  the  IMIS  Demonstration  System  was  GDE 
Systems,  with  its  subcontractors  Applied  Science  Associates,  Inc.,  Softech,  and  Systems  Control 
Technology.  In  addition,  technical  support  was  provided  to  the  Laboratory  by  NCI  Information 
Systems,  the  University  of  Dayton  Research  Institute,  Computer  Sciences  Corporation,  RJO 
Enterprises,  and  Robins-Gioia,  Inc.  Additional  support  was  received  from  the  Lockheed 
Corporation,  under  the  sponsorship  of  the  F-16  System  Program  Office. 

The  user  demonstration  and  field  test  could  not  have  been  accomplished  without  the 
generous  support  of  the  host  unit,  the  3 10th  Fighter  Squadron.  The  efforts  of  Capts.  Tribble  and 
Ramero,  CMSgt  Rios,  and  the  technicians  and  supervisors  under  their  command  were  a  major 
factor  in  the  success  of  the  field  test. 


INTEGRATED  MAINTENANCE  INFORMATION  SYSTEM; 
DEMONSTRATION  AND  FIELD  TEST 


SUMMARY 

This  report  summarizes  the  results  of  the  final  phase  of  a  program  to  develop  and 
evaluate  the  concept  of  an  Integrated  Maintenance  Information  System  (IMIS).  IMIS  will 
provide  maintenance  technicians  and  managers  with  the  capability  to  access  all  the  technical 
information  required  to  perform  their  jobs  via  a  single,  integrated  system.  The  project  developed 
and  field  tested  the  technologies  to  implement  the  concept.  The  technologies  were  then 
implemented  in  a  demonstration  system  for  evaluation.  Technological  developments  include 
advances  in  the  areas  of  interactive  electronic  technical  manuals  (lETMs);  interactive,  computer¬ 
generated  diagnostics;  human/computer  interface;  special  purpose  portable  computers;  and 
integration  of  divergent  data  bases.  An  IMIS  demonstration  system,  applying  these  technologies, 
was  developed  and  evaluated. 

The  IMIS  program  was  established  to  evaluate  the  IMIS  concept  and  to  develop  and  test 
the  technology  required  for  an  operational  IMIS.  Major  activities  under  the  program  included: 

a.  Comprehensive  analysis  of  maintenance  information  requirements  to  serve  as  the 
baseline  for  developing  IMIS; 

b.  Research  to  develop  an  advanced,  interactive,  diagnostic-aiding  capability; 

c.  Research  to  develop  effective  human/computer  interface  and  technical 
information  presentation  techniques  to  make  IMIS  easy  to  use  and  an  effective 
means  of  communicating  technical  information; 

d.  Development  of  a  methodology  for  authoring  and  coding  lETMs  (technical 
orders  [TOs])  in  an  efficient  and  cost  effective  manner,  which  provides  flexibility 
and  reduces  redundancy; 

e.  Development  of  demonstration  portable  maintenance  aids  (PMAs)  using  off-the- 
shelf  components;  and 

f  Development  of  an  IMIS  demonstration  system  which  incorporates  the  major 
features  and  fimctions  of  IMIS  for  use  in  evaluating  IMIS  technology  and 
validating  requirements. 

The  IMIS  Demonstration  System  was  evaluated  in  a  three-phase  field  test  at  Luke  Air 
Force  Base  (AFB),  AZ,  in  the  spring  and  summer  of  1994.  This  report  summarizes  the  findings 
of  the  third  phase  of  the  evaluation,  a  test  to  evaluate  the  ability  of  the  system  to  support  on- 
aircraft  fault  isolation  and  repair  tasks.  In  the  test,  an  experiment  was  conducted  to  evaluate  the 
impact  of  IMIS  on  the  performance  of  maintenance  technicians.  In  the  experiment,  12  avionics 
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specialists  and  12  non-specialists  (airplane  general  [APG]  technicians)  performed  12  fault 
isolation  problems  on  three  F-16  subsystems:  the  fire  control  radar  (FCR),  heads-up  display 
(HUD),  and  inertial  navigation  system  (INS).  Half  the  problems  were  performed  using  the 
current  paper-based  TOs  and  part-ordering  and  documentation  procedures,  and  half  were 
performed  using  the  IMIS.  The  APG  technicians  were  included  in  the  study  to  determine  if  the 
use  of  IMIS  would  enable  non-specialist  technicians,  with  little  or  no  training  on  a  specific 
aircraft  subsystem,  to  isolate  and  repair  faults  in  that  system  at  least  as  effectively  as  specialists, 
with  specific  experience  and  training  on  the  system,  using  paper  TOs. 

Analysis  of  the  technicians’  performance  on  the  12  test  problems  suggests  the  following 
benefits  from  IMIS. 

a.  The  rate  for  successful  problem  completion  was  improved  by  approximately  22 
percent  for  specialists  and  42  percent  for  APG  technicians. 

b.  The  frequency  of  serious  errors  (errors  which  could  lead  to  failure  to  correct  the 
problem)  were  reduced  by  58  percent  for  specialists  and  83  percent  for  APG 
technicians. 

c.  Overall  problem  times  (i.e.,  troubleshoot,  order  parts,  repair,  and  document)  were 
reduced  by  approximately  17  percent  for  specialists  and  29  percent  for  APG 
technicians.  Analysis  indicates  that  the  reductions  in  times  were  due  primarily  to 
time  saved  by  the  automated  part-ordering  and  work  order  close-out  features  of 
IMIS. 

d.  Overall  parts  consumption  for  the  12  problems  was  reduced  by  26  percent  for 
specialists  and  37  percent  for  APG  technicians.  In-depth  analysis  indicates  that 
IMIS  diagnostics  are  much  more  beneficial  for  the  more  complex  problems  (e.g., 
most  of  the  part  savings  were  from  the  INS  system). 

e.  Part  ordering  times  were  reduced  by  94  percent  for  both  specialists  and  APG 
technicians. 

f  For  all  measures,  the  performance  of  the  APG  technicians  when  using  IMIS  was 
approximately  equal  to  the  performance  of  the  specialists  when  using  paper  TOs. 

A  key  finding  is  that  the  use  of  IMIS  in  effect  “leveled  the  playing  field”  for  the  APG 
technicians.  With  IMIS,  the  APG  technicians  were  able  to  perform  the  fault  isolation  problems 
as  effectively  as  specialists  using  IMIS  and  more  effectively  than  the  specialists  using  paper  TOs. 
This  finding  suggests  that  the  use  of  IMIS  will  enable  the  crew  chief  to  perform  a  much  wider 
range  of  tasks,  thus  reducing  reliance  on  specialists. 

The  responses  to  questionnaires  administered  to  the  technicians  after  completing  the  test 
problems  indicate  a  high  rate  of  acceptance  of  IMIS  and  a  desire  to  see  it  implemented  in  the  Air 
Force.  IMIS  features  which  were  rated  highly  include  the  integration  of  technical  data. 
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automated  parts-ordering  by  radio  frequency  (RF)  link,  automated  work  order  close-out,  and 
automated  diagnostic  advice. 


INTRODUCTION 

This  report  presents  the  results  of  the  final  phase  of  a  program  to  develop  and  evaluate 
the  concept  of  an  IMIS.  An  IMIS  will  provide  the  maintenance  technician  with  the  capability  to 
access  all  technical  information  (lETMs,  interactive  diagnostics  instructions,  work  orders,  supply 
availability  and  ordering,  historical  data,  training  material,  etc.)  required  to  maintain  aircraft,  via 
a  single,  integrated  system,  regardless  of  the  source  of  that  information.  This  capability  will  help 
maintain  the  new  technology  found  in  advanced  weapon  systems  and  will  also  support  the 
current  trend  toward  fewer  maintenance  specialties.  In  the  future,  portable  computers  will 
deliver  the  electronic  data  at  the  flight  line  work  site,  and  a  network  of  work  stations  will  interact 
with  existing  maintenance  computer  systems  in  shops  and  work  centers.  The  IMIS  is  designed 
to  be  totally  integrated  with  other  maintenance  information  systems  during  peacetime  at  main 
operating  bases,  but  the  basic  portable  diagnostic  and  TO  automation  is  fially  deployable  to 
remote  dispersed  locations  during  war  (Johnson,  1994).  The  project  developed  and  field  tested 
the  technology  to  implement  the  concept.  The  technologies  were  then  implemented  in  a 
demonstration  system  for  evaluation. 


BACKGROUND 

The  IMIS  program  was  an  advanced  development  demonstration  project  conducted  by 
the  AL/HRGO.  The  program  was  established  to  evaluate  the  IMIS  concept  and  to  develop  and 
test  the  technology  required  for  an  operational  IMIS.  Major  activities  under  the  program 
included: 

a.  A  comprehensive  analysis  of  maintenance  information  requirements  to  serve  as 
the  baseline  for  developing  an  IMIS; 

b.  Research  to  develop  an  advanced,  interactive,  diagnostic-aiding  capability; 

c.  Research  to  develop  effective  human/computer  interface  and  technical 
information  presentation  techniques  to  make  the  IMIS  easy  to  use  and  an 
effective  means  of  communicating  technical  information; 

d.  Development  of  a  methodology  for  authoring  and  coding,  interactive  electronic 
maintenance  technical  data  (TOs)  in  an  efficient  and  cost-effective  manner  that 
provides  flexibility  and  reduces  redundancy; 

e.  Development  of  demonstration  PMAs  using  off-the-shelf  components;  and 

f  Development  of  a  demonstration  system  which  incorporates  the  major  features 
and  functions  of  an  IMIS  for  use  in  evaluating  the  technology  and  for  validating 
requirements. 
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Reports  and  publications  documenting  the  research  conducted  under  the  IMIS  program  are  listed 
in  a  comprehensive  bibliography  at  the  end  of  this  report. 

A  full  implementation  of  the  IMIS  concept  will  have  the  following  features  and 
capabilities  (Johnson,  1994); 

•  BETMs  —  TOs  in  digital  form,  presented  on  a  PMA. 

•  Interactive  diagnostic-aiding  via  a  PMA,  supported  by  an  interface  with  the 
aircraft  data  bus  to  operate  built-in-tests  (BITs)  and  download  subsystem 
performance  data  for  use  by  the  IMIS  maintenance  diagnostic  algorithm. 

•  Interface  with  maintenance  data  collection  systems  such  as  the  Core  Automated 
Maintenance  System  (CAMS),  Standard  Base  Supply  System  (SBSS),  and  other 
information  systems  and  data  bases  which  support  maintenance  operations. 

•  Automated  maintenance  data  collection  and  reporting,  via  the  CAMS  interface. 

•  Training  materials  integrated  with  the  lETM  and  automated  diagnostics  data  base 
to  support  on-the-job  and  upgrade  training. 

•  IMIS  local  area  network  (LAN),  connects  IMIS  workstations,  RF  modems, 
printers,  data  storage  devices,  and  interfaces  with  external  data  bases. 

•  Communication  capability  via  RF  link  between  PMAs  and  the  IMIS  LAN. 

A  fiilly  implemented  IMIS  would  include  the  following  basic  hardware  and  software: 

•  PMAs  —  special  purpose  computers,  four  to  six  pounds,  fully  ruggedized, 
multiple  power  sources,  removable  hard  disk  drive,  screen  readable  under  all 
conditions,  self-contained  batteries,  interface  to  aircraft  maintenance  bus,  usable 
with  chemical  and  cold  weather  gloves,  easy  to  operate  without  training,  keypad, 
does  not  require  typing  skills. 

•  Maintenance  Information  Workstations  (MIWs)  —  desktop  computers  to  support 
maintenance  management  activities,  maintain  the  IMIS  data  base,  and  interface 
with  external  data  bases.  Located  in  maintenance  management  and  support 
offices. 

•  Mobile  Workstation  Computers  —  RF  capability,  mounted  in  maintenance 
supervisory  vehicles  for  use  in  managing  and  controlling  flightline  maintenance 
activities. 
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Technical  Data  Presentation  Software  —  presents  lETM  data  and  diagnostic 
procedures. 

Interactive  Diagnostics  Software  —  generates  most  efficient  diagnostic  strategy 
based  upon  technical  data,  system  design  information,  component  failure  rates, 
and  current  system  status  (e.g.,  symptoms  and  test  results). 

IMIS  Data  Base  —  maintains  data  required  to  support  the  IMIS  and  maintains  a 
backup  copy  of  the  relevant  CAMS  information  to  prevent  interruption  of 
operations  if  CAMS  is  down. 

External  Data  Base  Interface  Software  —  controls  interface  with  CAMS,  SBSS, 
and  other  external  data  bases. 


IMIS  REQUIREMENTS  ANALYSIS  AND  DEMONSTRATION 
SYSTEM  DEVELOPMENT 

The  final  phase  of  the  IMIS  program  was  a  large  contract  effort  to  define  requirements 
and  specifications  for  an  operational  IMIS,  develop  a  demonstration  system  incorporating  the 
major  functions  and  capabilities  of  an  IMIS,  evaluate  the  IMIS  concept  using  the  demonstration 
system,  and  update  the  specifications  based  upon  lessons  learned  in  the  demonstration  system 
evaluation.  The  major  activities  under  this  effort  are  summarized  below. 

A  comprehensive  analysis  was  conducted  to  define  information  requirements  to  support 
maintenance  of  tactical  fighter  aircraft.  The  analysis  was  accomplished  by  interviewing 
maintenance  technicians  and  supervisors,  and  by  observing  maintenance  operations  at  nine 
Tactical  Air  Command  and  United  States  Air  Forces  -  Europe  bases,  interviewing  personnel 
responsible  for  developing  and  maintaining  existing  maintenance  data  systems  (e.g.,  CAMS)  at 
the  Air  Force  Systems  Design  Center,  interviewing  senior  Air  Force  maintenance  managers,  and 
reviewing  Air  Force  regulations.  Integrated  Computer-Aided  Manufacturing  (ICAM)  Definition 
(IDEF)  modeling  techniques  were  then  used  to  develop  "AS  IS"  and  "TO  BE"  models  of  the 
information  flow  in  a  fighter  squadron  (FS)  maintenance  organization. 

The  TO  BE  model  defined  the  sources,  processing,  transmission,  and  uses  of  information 
within  a  maintenance  organization  when  supported  by  IMIS.  The  model  was  based  upon  the  AS 
IS  model,  the  IMIS  operational  concept  document,  a  review  of  applicable  hardware  and  software 
technology,  and  the  inputs  and  recommendations  of  maintenance  analysts,  software  engineers, 
computer  engineers,  and  information  management  specialists.  The  AS  IS  and  TO  BE  models  are 
defined  and  described  in  the  IMIS  Architecture  Document  (General  Dynamics  Electronics 
Division,  1990). 

The  IMIS  Architecture  provided  the  basis  for  the  development  of  the  IMIS  System 
Segment  Specification  (SSS).  The  SSS  established  system  design,  performance  requirements, 
design  goals,  and  test  requirements  for  use  in  developing  an  operational  IMIS.  In  addition  to 
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basic  design  requirements,  the  SSS  provided  a  comprehensive  listing  of  the  functions  the  IMIS 
must  perform  and  the  performance  criteria  that  must  be  met.  A  high-level  representation  of  the 
IMIS  architecture  is  presented  in  Figure  1 . 

The  functional  requirements  for  an  IMIS  specified  in  the  SSS  were  prioritized  to  develop 
a  set  of  essential  functions  for  evaluation.  The  IMIS  Demonstration  System  was  then  designed 
and  developed  to  provide  these  functions  for  evaluation  of  the  IMIS  concept  and  to  provide  data 
on  the  impact  of  the  IMIS  on  the  performance  of  maintenance  technicians  and  the  overall 
operation  of  the  maintenance  organization.  The  IMIS  Demonstration  System  implemented  the 
basic  architecture  specified  in  the  SSS,  except  that  all  functions  were  not  provided.  A  PMA  was 
designed  and  fabricated,  and  an  off-the-shelf  workstation  was  selected  to  serve  as  the  MIW.  A 
mockup  of  the  aircraft  interface  panel  (AIP)  was  developed  in  place  of  developing  an  actual  AIP 
for  the  F-16  aircraft  (due  to  cost  considerations).  The  Demonstration  System  PMA  is  presented 
in  Figure  2. 


IMIS  FIELD  TEST 

The  IMIS  Demonstration  System  was  evaluated  in  a  three-phase  field  test  at  Luke  AFB, 
AZ  in  the  Spring  and  Summer  of  1994.  In  Phase  I,  the  IMIS  debriefing  function  was  evaluated. 
The  results  of  this  evaluation  indicated  that  the  IMIS  provides  an  effective  debriefing  capability. 
Use  of  the  system  resulted  in  faster  debriefs  for  Code  1  aircraft.  In  addition,  it  has  the  capability 
to  provide  the  technician  with  more  extensive  information  on  reported  system  failures  than 
provided  by  current  debriefing  procedures.*  Debriefers  who  used  the  system  found  it  effective 
and  easy  to  use  (Ward,  Weimer,  and  Kruzick  1995b). 

The  Phase  II  evaluation  was  an  "End-to-End"  Demonstration  designed  to  evaluate  the 
capability  of  the  IMIS  to  support  flightline  management  functions,  especially  those  performed  by 
the  Production  Superintendent  and  Expediters.  To  avoid  interfering  with  ongoing  maintenance 
operations,  the  evaluation  was  conducted  in  a  classroom  environment.  Expediters  and 
Production  Superintendents  used  the  IMIS  to  perform  typical  scenarios  incorporating  the  tasks 
they  normally  perform  in  their  day-to-day  activities.  These  included  performing  actions  such  as 
assigning  a  work  order  to  a  technician,  creating  and  closing  work  orders,  approving  part  orders, 
retrieving  aircraft  status  information,  changing  aircraft  status,  and  sending  messages.  The 
participants  were  asked  to  evaluate  the  system  as  an  aid  for  performing  their  jobs.  The 
evaluations  were  very  positive.  See  Ward,  Kruzick,  and  Weimer  (1995c)  for  details  of  the  end- 
to-end  demonstration. 


*  This  benefit  was  not  fully  evaluated  because  the  test  was  conducted  under  operational 
conditions  and  the  current  paper-based  system  has  no  mechanism  available  to  provide  this 
information  to  the  technician.  In  a  full  implementation  of  the  IMIS,  this  information  would  be 
automatically  entered  into  the  IMIS  data  base,  loaded  onto  the  technician’s  PMA,  and  used  by 
the  diagnostic  routine  to  isolate  the  fault. 
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Module 


Joint  Continuous  Acquisition  and  Lifecycle  Support 
'Centralized  Engine  Monitoring  System 

Figure  1 .  High-Level  IMIS  Architecture 


Figure  2.  Demonstration  System  Portable  Maintenance  Aid 


Finally,  the  Phase  III  evaluation  (Field  Test)  was  designed  to  evaluate  the  capability  of 
the  IMIS  to  support  on-aircraft  maintenance,  especially  diagnostics.  The  results  of  this  test  are 
described  in  the  remainder  of  this  report. 

Objectives 

The  IMIS  Field  Test  was  conducted  to  evaluate  the  ability  of  the  IMIS  to  support  on- 
aircraft  maintenance  —  especially  diagnostic  tasks  —  and  to  provide  additional  data  for  use  in 
evaluating  the  overall  effectiveness  of  the  IMIS.  The  Field  Test  had  the  following  the 
objectives: 

•  Demonstrate  that  use  of  the  IMIS  can  significantly  improve  the  troubleshooting 
performance  of  technicians  by  reducing  the  time  required  to  return  an  aircraft  to 
service  and  reducing  the  number  of  errors  made  by  technicians. 

•  Demonstrate  that  use  of  the  IMIS  can  reduce  the  consumption  of  spare  parts  by 
reducing  the  number  of  good  parts  unnecessarily  replaced  during  the  diagnostic 
process  (i.e.,  reduce  the  retest  OK  rate). 

•  Demonstrate  that  technicians  without  extensive,  specialized  training  on  a  given 
system  (e.g.,  airplane  general  [APG]  technicians)  can  effectively  perform 
troubleshooting  tasks  on  the  system  when  using  the  IMIS. 

•  Demonstrate  that  use  of  the  IMIS  will  reduce  the  time  required  to  document 
maintenance  actions  taken  and  close-out  work  orders. 

•  Demonstrate  that  use  of  the  IMIS  will  provide  more  accurate  information  on 
maintenance  actions  taken  for  input  to  the  Maintenance  Data  Collection  system. 

•  Develop  data  to  demonstrate  the  technical  and  cost  advantages  of  the  IMIS  over 
the  current  paper-based  system. 

•  Identify  the  strengths  and  weaknesses  of  the  IMIS,  identify  changes  to  the  IMIS 
which  would  make  it  more  effective,  and  identify  changes  to  the  IMIS 
specifications  required  for  an  operational  implementation  of  the  IMIS. 

Hypotheses 

The  Field  Test  was  designed  to  test  the  following  hypotheses. 

1.  Avionics  specialist  technicians  will  successfully  complete  significantly  more 

diagnostic  problems  when  using  the  IMIS  as  the  source  of  technical  information 
than  when  using  the  paper  TO. 
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2.  APG  technicians  using  the  IMIS  as  the  source  of  technical  information  will 
successfully  complete  significantly  more  diagnostic  problems  than  will  avionics 
specialists  using  paper  TOs  as  their  source  of  technical  information. 

3.  Avionics  specialist  technicians  using  the  IMIS  to  perform  troubleshooting  tasks 
will  replace  significantly  fewer  parts  when  using  the  IMIS  than  when  using  paper 
TOs. 

4.  APG  technicians  using  the  IMIS  to  perform  troubleshooting  tasks  will  replace 
significantly  fewer  parts  than  will  avionics  specialists  using  paper  TOs. 

5.  Avionics  specialist  technicians  will  make  significantly  fewer  maintenance  errors 
while  performing  the  troubleshooting  problems  when  using  the  IMIS  than  when 
using  paper  TOs. 

6.  APG  technicians  will  make  significantly  fewer  maintenance  errors  while 
performing  the  troubleshooting  problems  when  using  the  IMIS  than  will  avionics 
specialists  using  the  paper  TOs. 

7.  Avionics  specialist  technicians  will  require  significantly  less  total  time  to  complete 
troubleshooting  problems  when  using  the  IMIS  than  when  using  paper  TOs. 

8.  APG  technicians  will  require  significantly  less  total  time  to  complete  the 
troubleshooting  problems  when  using  the  IMIS  than  will  avionics  specialists  using 
the  paper  TO. 

9.  Avionics  specialist  technicians  will  require  significantly  less  time  to  prepare  for  a 
troubleshooting  task  when  using  the  IMIS  as  the  source  of  technical  information 
than  when  using  the  paper  TO. 

10.  APG  technicians  using  the  IMIS  will  require  significantly  less  time  to  prepare  for 
a  troubleshooting  task  than  will  avionics  specialists  using  paper  TOs. 

1 1 .  Avionics  specialist  technicians  will  require  significantly  less  time  to  verify  faults  in 
the  test-bed  subsystems  when  using  the  IMIS  than  when  using  paper  TOs. 

12.  APG  technicians  using  the  IMIS  will  require  significantly  less  time  to  verify  faults 
in  the  test-bed  subsystems  than  will  avionics  specialists  using  paper  TOs. 

13.  Avionics  specialist  technicians  will  require  significantly  less  time  to  isolate  faults 
in  the  test-bed  subsystems  when  using  the  IMIS  than  when  using  paper  TOs. 

14.  APG  technicians  using  the  IMIS  will  require  significantly  less  time  to  isolate  faults 
in  the  test-bed  subsystems  than  will  avionics  specialists  using  paper  TOs. 
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15.  Avionics  specialist  technicians  will  require  significantly  less  time  to  order  each 
part  when  using  the  IMIS  than  when  using  current  part  ordering  procedures. 

16.  APG  technicians  using  the  IMIS  will  require  significantly  less  time  to  order  each 
part  than  required  by  avionics  specialists  using  current  part  ordering  procedures. 

17.  Avionics  specialist  technicians  will  require  significantly  less  time  to  complete 
required  documentation  when  using  the  IMIS  than  when  using  current 
documentation  and  close-out  procedures. 

18.  APG  technicians  using  the  IMIS  will  require  significantly  less  time  to  complete 
required  documentation  than  will  avionics  specialists  using  current  documentation 
and  close-out  procedures. 

19.  Avionics  specialist  technicians  will  make  significantly  fewer  documentation  errors 
when  using  the  IMIS  than  when  using  current  documentation  procedures. 

20.  APG  technicians  using  the  IMIS  will  make  significantly  fewer  documentation 
errors  than  will  avionics  specialist  technicians  using  current  documentation 
procedures. 


METHOD 

The  objective  of  this  effort  was  to  compare  the  performance  of  technicians  using  the 
EMIS  with  their  performance  using  paper  TOs  and  to  compare  the  performance  of  APG 
technicians  using  the  IMIS  with  the  performance  of  specialists;  therefore,  it  was  essential  that 
performance  of  the  technicians  be  measured  under  comparable  conditions.  Consequently,  it  was 
necessary  to  measure  the  performance  of  technicians  engaged  in  very  similar  tasks  under  very 
similar  conditions,  preferably  doing  the  same  tasks  under  identical  conditions.  To  ensure  the 
required  symmetry  of  tasks  and  data  collection  conditions,  an  experimental  approach  was 
adopted. 

An  experiment  was  conducted  in  which  specialist  and  APG  technicians  performed  12 
troubleshooting  tasks,  six  using  the  IMIS  and  six  using  paper  TOs.  The  order  of  task 
presentation  was  counterbalanced  so  that  half  the  technicians  performed  their  IMIS  tasks  first 
and  half  performed  their  paper-based  TO  tasks  first.  Twelve  troubleshooting  tasks  for  three 
F-16  subsystems  were  used  as  test  problems  for  the  study.  The  test-bed  subsystems  were  the 
F-16  FCR,  HUD,  and  INS.  The  tasks  were  selected  in  pairs  so  that  paired  tasks  were 
approximately  equal  in  difficulty  and  required  skills.  The  tasks  were  representative  of  the 
troubleshooting  tasks  normally  encountered  in  maintaining  the  test-bed  subsystems. 
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Experimental  Design 


A  two-factor  experiment  with  repeated  measure  measures  was  used  for  the  study  (after 
Winer,  Brown,  &  Michels,  1991)  for  all  performance  measures  except  parts  used.  The  factors 
were  specialty  (avionics  specialist  vs.  APG  technician)  and  media  (IMIS  vs.  paper  TO). 

Repeated  measures  were  made  on  the  media  factor,  with  each  subject  performing  half  the  tasks 
with  IMIS  and  half  with  paper  TOs.  Specialty  was  measured  between  groups.  Problem  order 
within  the  blocks  of  tasks  was  systematically  varied  to  avoid  order  effects  and  to  ensure  that  each 
task  was  performed  an  equal  number  of  times  with  IMIS  and  with  paper  TOs.  The  experimental 
design  and  problem  administration  sequence  are  presented  in  Appendix  A. 

Subjects 

Twelve  avionics  technicians  (specialists)  and  12  APG  technicians  served  as  subjects  for 
the  study.  Eight  of  the  avionics  technicians  were  from  the  310  FS,  three  from  the  63  FS,  and  one 
from  the  308  FS.  Eight  were  Skill  Level  5  and  four  were  Skill  Level  3.  Their  experience  levels 
ranged  from  16  months  to  108  months.  The  12  APG  technicians  were  all  from  the  3 10  FS.  Two 
were  Skill  Level  5  and  ten  were  Skill  Level  3.  Their  experience  levels  in  F-16  maintenance 
ranged  from  two  months  to  70  months.  All  subjects  were  male  and  were  assigned  to  the  project 
by  their  supervisors. 


Test  Facilities 

The  field  test  was  conducted  using  the  facilities  of  the  310  FS,  Luke  AFB,  AZ.  The 
squadron  provided  an  aircraft,  hangar  space  for  the  aircraft,  office/training  room,  required  test 
equipment,  ground  support  equipment,  and  tools.  The  office  was  located  in  a  hangar.  For  most 
of  the  test  period,  the  aircraft  was  parked  in  the  same  hangar,  approximately  50  feet  from  the 
office  door.  The  remainder  of  the  time,  it  was  parked  in  one  of  two  adjacent  hangars,  one 
approximately  150  feet  from  the  office  and  the  other  approximately  200  feet  from  the  office. 

The  assigned  aircraft  was  dedicated  to  the  test.  No  maintenance  or  other  actions  were 
performed  on  the  aircraft  during  the  assigned  period.  The  aircraft  was  changed  periodically  to 
meet  squadron  mission  requirements.  Normally,  an  aircraft  was  assigned  for  a  one-week  period. 
An  intercom  system  was  used  for  communications  between  the  technician  and  data  collectors 
during  periods  of  high  noise  (e.g.,  when  the  ground  power  and  air  conditioning  units  were 
operating). 

In  addition  to  administrative  functions,  the  IMIS  office  was  used  for  technician 
orientation,  training,  as  technician  break  room,  and  as  a  staging  area  for  problem  initiation.  Also, 
a  file  of  TOs  required  for  the  test  problems  was  maintained  in  the  office. 
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IMIS  Demonstration  System 


The  IMIS  Demonstration  System,  as  installed  at  Luke  AFB,  served  as  the  test  system  for 
the  Field  Test.  The  demonstration  system  installed  at  Luke  AFB  included  the  following 
equipment: 

1  MIW  installed  in  the  3 10  FS  Debriefing  Room 
1  MIW  installed  in  the  Combat  Oriented  Support  Organization  (COSO) 

1  MIW  installed  in  the  IMIS  Project  Office 

1  memory  module  loading  station  installed  in  the  IMIS  Project  Office 
6  PMAs  maintained  in  the  IMIS  Project  Office 

2  Laptop  computers  which  served  as  mobile  MIWs  for  the  Expediters  and  Production 

Superintendent 

1  RF  modem  installed  on  the  310  FS  hangar 
1  RF  modem  installed  above  the  IMIS  Project  Office 

The  MIWs  were  connected  via  a  fiber  optic  cable  to  form  the  IMIS  LAN  and  were 
interfaced  with  the  CAMS  network.  The  following  capabilities  of  the  IMIS  Demonstration 
System  were  used,  to  some  extent,  during  the  Field  Test. 

PMA  removable  hard  disk  drive  loading 
Presentation  of  work  orders 
Presentation  of  TOs 
Diagnostics  module 
1553  interface 

Parts  ordering/part  approval  process  (via  RF  link) 

Work  order  close-out/documentation 

The  PMA  removable  disk  drive  loading  capability  was  used  to  load  data  onto  the  disk 
drives  with  the  proper  work  orders  and  technical  data  for  presentation  on  the  PMA.  The 
removable  disk  drive  loader  was  used  by  the  data  collection  staff  only. 

Job  Performance  Tests 

Job  performance  tests  were  used  to  measure  technician  performance.  Each  job 
performance  test  measured  the  technician’s  performance  on  one  diagnostic  problem  — 
identifying  and  correcting  the  cause  of  an  observed  fault  in  the  system.  Job  performance  tests 
were  developed  for  12  diagnostics  problems.  The  test  problems  were  based  on  12  faults  judged 
to  be  typical  of  those  occurring  in  the  FCR,  HUD,  and  INS  systems.  Four  tests  were  developed 
for  each  subsystem;  these  tests  were  developed  as  "matched  pairs."  The  two  tests  which 
compose  a  pair  were  designed  to  be  as  nearly  equal  as  possible  in  terms  of  difficulty,  time  to 
perform,  and  number  of  subtasks  (e.g.,  diagnostic  tests  performed)  as  possible.  Comparability  of 
test  problems  was  accomplished  by  selecting  two  different  fault  codes  (returned  by  the  aircraft 
BIT)  or  malfunction  symptoms  that  led  to  similar  maintenance  actions  (e.g.,  replace  the  same 
component  or  repair  different  wires  in  the  same  wire  bundle).  Troubleshooting  the  two  fault 


12 


codes  followed  almost  identical  troubleshooting  procedures,  typically  deviating  only  at  the  last 
step  or  two 

Test  problems  were  selected  based  upon  several  criteria.  These  selection  criteria  were; 

a.  It  must  be  possible  to  simulate  the  problem  without  damaging  the  aircraft  or 
causing  a  safety  hazard. 

b.  The  test  problem  must  be  representative  of  faults  normally  encountered  in  the 
day-to-day  maintenance  of  the  subsystem. 

c.  The  problem  must  be  sufficiently  difficult  to  test  the  capabilities  of  the  IMIS  and 
paper-based  TO  troubleshooting  aids. 

d.  The  problem  must  not  require  more  than  approximately  one  hour  to  complete 
(under  test  conditions). 

e.  It  must  be  possible  to  develop  a  companion  problem  of  approximately  equal 
difficulty. 

A  large  number  of  subsystem  faults  were  evaluated  in  selecting  the  test  problems.  It  was 
found  that  very  few  problems  could  satisfy  all  the  above  criteria.  The  faults  used  were  the  best 
of  those.  The  twelve  faults  used  and  their  associated  maintenance  fault  list  (MFL)  codes  are 
listed  in  Table  1. 

Breakout  boxes  were  used  to  simulate  faults  used  for  the  test  problems.  The  breakout 
boxes  were  installed  between  line  replaceable  units  (LRUs)  of  the  test  subsystem.  The  breakout 
boxes  provide  a  means  of  interrupting  communication  between  the  LRUs  by  breaking  the 
continuity  of  selected  wire  or  wires.  Thus,  it  is  possible  to  simulate  a  malfunction  in  an  LRU  by 
interrupting  its  output  to  another  component  of  the  subsystem.  This  produces  the  same 
symptoms  as  if  the  LRU  were  actually  defective. 

None  of  the  available  tests  could  be  performed  in  their  "entirety"  within  the  time 
constraint,  primarily  because  of  the  time  required  to  perform  several  repetitive  tasks.  For 
example,  several  problems  require  alignment  of  the  INS,  either  to  verify  system  health  or  as  part 
of  the  troubleshooting  process.  This  is  a  lengthy  automated  process  with  little  input  from  the 
technician.  The  time  required  to  perform  the  alignment  is  the  same  when  using  the  IMIS  as 
when  using  the  paper  TO.  Thus,  it  consumes  time  without  providing  any  meaningful  information 
for  test  purposes.  Similar  situations  exist  for  the  FCR  and  HUD.  To  conserve  testing  time, 
standard  times  were  used  for  these  subtasks.  This  was  done  by  the  observer  telling  the 
technician  to  assume  that  the  task  had  already  been  done. 


2  The  similarity  of  the  problems  was  not  readily  apparent  to  the  technicians  because  they  were 
working  with  different  technical  data  (one  with  IMIS  the  other  with  the  TO). 
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Table  1.  Faults  Used  for  Job  Performance  Tests 


Problem 

Number 

Subsystem 

Fault  Code 

Symptom/MFL 

Fault 

FI 

FCR 

AD 

MFL  001 

Wiring 

F2 

FCR 

AE 

MFL  002 

Wiring 

F3 

FCR 

AN 

MFL  004 

PSP 

F4 

FCR 

AU 

MFL  084 

PSP 

H5 

HUD 

AD 

MFL  001 

EU 

H6 

HUD 

AE 

MFL  002 

EU 

H7 

HUD 

BM 

Depr.  Ret.  Sw. 

Wiring 

H8 

HUD 

BR 

Test  Sw. 

Wiring 

19 

INS 

AG 

MFL  on 

Wiring 

no 

INS 

AG 

MFL  Oil 

Wiring 

Ill 

INS 

AG 

MFL  Oil 

HSI 

112 

INS 

AG 

MFL  Oil 

HSI 

(Note:  Although  problems  9  through  12  all  have  the  same  fault  code  (AG)  and 
MFL  (01 1),  they  are  in  fact  different  problems.  Problems  9  and  10  both  lead  to 
wiring  repairs,  but  different  wires  are  bad.  Problems  1 1  and  12  both  lead  to 
replacement  of  the  HSI,  due  to  different  malfunctions  within  the  HSI.) 

Standard  times  were  used  in  four  situations:  (1)  for  tasks  that  were  too  time  consuming 
and  yielded  no  new  information  on  technician  performance  (e.g.,  FCR  pressurization  test 
required  following  antenna  replacement);  (2)  for  tasks  which  risked  damage  to  the  aircraft  (e,g., 
removal  of  the  HUD  pilot's  display  unit^  (3)  for  rectifications,  since  it  was  not  practical  to  make 
actual  repairs  (to  avoid  damage  to  the  aircraft  due  to  repeated  repairs);  and  (4)  for  the  final 
system  health  checks  to  verify  that  the  subsystem  is  fully  operational.  The  technicians  had 
already  performed  the  system  health  checks  as  part  of  the  problem  verification,  demonstrating 
the  capability  to  do  those  checks  using  the  IMTS  or  paper  TO.  Thus,  little  new  information 
would  be  gained  by  repeating  the  checks  to  establish  system  health.  Standard  times  (based  upon 
observation  of  the  task  being  performed  or  estimates  from  technicians  experienced  in  performing 
the  task)  were  used  in  calculating  test  times.  When  standard  times  were  used,  the  technician  was 
required  to  tell  the  technical  observer  what  task  he  would  perform  and  show  the  required 
technical  data  (in  the  TO  or  on  the  IMIS). 

In  addition,  standard  times  were  used  to  account  for  times  required  to  perform  two 
processes  under  the  paper-based  condition  which  could  not  be  accomplished  during  the  test 
without  interfering  with  the  host  unit’s  operations.  These  processes  were  parts  ordering  and 
entering  work  order  close-out  information  into  the  CAMS.  It  was  not  necessary  to  use  standard 
times  for  the  IMIS  condition  because  these  processes  were  automatically  performed  by  the  IMIS. 

Several  actions  involved  in  ordering  a  part  in  actual  maintenance  operations  were  not 
considered  in  this  test.  These  include  the  time  to  obtain  authorization  to  order  the  part,  the  time 
to  travel  to  the  COSO  to  place  the  part  order,  the  time  for  the  COSO  to  order  the  part,  and  the 
time  for  the  supply  system  to  process  and  deliver  the  part.  There  was  no  realistic  way  to 
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simulate  this  process;  therefore,  standard  times  were  used.  An  additional  15  minutes  per  part 
order  was  added  for  each  time  a  part  was  ordered  under  the  paper-based  condition.  This 
estimate  was  based  on  estimates  given  by  technicians,  supervisors,  and  supply  personnel.  The  15 
minutes  includes  10  minutes  to  secure  the  aircraft  and  travel  to  the  COSO  and  five  minutes  to 
process  the  order  at  the  COSO.  The  estimates  used  are  conservative.  Times  are  frequently 
much  longer  due  to  unpredictable  factors,  such  as  the  supply  clerk  being  out  of  the  office  and 
delays  in  getting  transportation  from  the  aircraft  to  the  COSO. 

There  was  no  practical  way  to  simulate  the  process  of  entering  the  work  order  close-out 
information  into  CAMS.  The  host  unit  CAMS  terminals  could  have  been  used  but  would  have 
resulted  in  putting  test  data  into  the  active  CAMS  data  base,  which  was  unacceptable  to  the 
CAMS  data  base  manager.  Thus,  a  standard  time  was  used.  An  additional  ten  minutes  was 
added  to  the  close-out  time  to  adjust  for  the  time  to  input  the  data  to  CAMS.  This  adjustment 
was  made  for  the  paper  condition  only.  The  estimate  includes  five  minutes  to  travel  to  the 
maintenance  office  and  five  minutes  to  input  the  data.  Time  spent  waiting  for  access  to  a  CAMS 
terminal  was  not  considered.  This  is  believed  to  be  a  very  conservative  estimate. 

The  test  problems  yielded  the  following  performance  measures. 

a.  Preparation  Time.  Preparation  time  is  the  time  from  problem  assignment  until 
start  of  fault  verification.  For  the  paper  TO  condition,  preparation  time  includes 
the  time  to  identify  and  sign  out  required  TOs,  time  to  move  to  the  aircraft,  time 
to  locate  required  instructions,  and  questions  for  the  pilot.  For  the  IMIS 
condition,  preparation  time  includes  the  time  to  move  to  the  aircraft  and  set  up  to 
start  fault  verification  (including  questions  for  the  pilot  and  connecting  PMA  to 
1553  interface)  and  prepare  to  start  fault  verification. 

b.  Fault  Verification  Time.  Fault  verification  time  is  the  time  to  conduct  the 
necessary  checks  to  verify  that  reported  malfunction  symptoms  are  present  in  the 
subsystem. 

c.  Fault  Isolation  Time.  Fault  isolation  time  is  the  time  required  to  identify  the  cause 
of  the  observed  subsystem  malfunction,  rectify  the  fault,  and  verify  that  the 
subsystem  is  fully  operational.  This  time  is  composed  of  the  elapsed  time  from 
the  start  of  the  troubleshooting  checks  until  the  completion  of  the  system  health 
checks  to  verify  repair,  in  addition  to  standard  times  for  replacing  any  parts  or 
making  any  repairs,  ordering  parts,  and  completing  the  final  system  health 
checks.  3 


3  In  some  cases,  the  diagnostic  procedures  required  replacement  of  a  nondefective  part  as  a 
means  of  eliminating  that  component  from  the  set  of  possible  faults.  In  these  cases,  the  system 
health  checks  required  to  determine  if  the  fault  had  been  corrected  were  actually  performed  and 
times  were  recorded.  Standard  times  were  not  used. 
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d.  Close-Out  Time.  Close-out  time  is  the  time  necessary  for  the  technician  to 
complete  the  close-out  documentation.  For  the  IMIS  condition,  close-out  time  is 
the  time  to  enter  the  information  into  the  IMIS  using  the  PMA.  For  the  paper 
condition,  it  is  the  time  to  complete  a  paper  form  by  entering  the  required 
information  in  addition  to  a  standard  time  for  entering  the  information  into 
CAMS. 

e.  Problem  Performance  Time.  Problem  performance  time  is  the  total  time  required 
to  complete  the  test,  adjusted  for  any  necessary  time  outs  (due  to  external 
interruptions  such  as  an  unexpected  problem  with  the  aircraft). 

£  Total  Problem  Time.  Total  problem  time  is  the  problem  performance  time  and 

the  standard  times  for  performing  rectifications,  conducting  system  health  checks, 
and  completing  documentation. 

g.  Successful  Fault  Isolation.  Successful  fault  isolation  indicates  whether  the 
technician  successfiilly  identified  the  fault  and  verified  that  the  subsystem  was 
fully  operational.^ 

h.  Number  of  Parts  Used.  The  “number  of  parts  used”  measure  indicates  the 
number  of  parts  the  technician  used  to  rectify  the  problem  (including  both 
necessary  and  unnecessary  replacements). 

i.  Parts  Ordering  Time.  Parts  ordering  time  is  the  time  required  to  order  parts.  For 
the  IMIS  condition,  it  is  the  time  required  for  the  technician  to  enter  the  parts 
order  request  into  the  PMA.  For  the  paper  condition,  it  is  the  time  to  locate  the 
required  parts  ordering  information  and  fill  out  the  parts  order  form  plus  a 
standard  time  of  15  minutes  for  submitting  and  processing  the  order. 

j.  Errors  Made.  The  “errors  made”  measure  refers  to  the  number  of  errors  due  to 
incorrect  use  of  IMIS  or  the  paper  TO,  the  number  of  maintenance  errors  (e.g.,  a 
technician  taking  a  continuity  reading  from  wrong  connector  pin),  and  the  number 
of  documentation  errors. 

k.  Helps  Given.  “Helps  given”  indicates  the  number  of  assists  provided  on  the  use 
of  IMIS  or  paper  TOs  and  the  number  of  system  knowledge  assists  (help  in 
locating  a  component).^ 


^  Since  the  final  system  health  was  simulated,  the  critical  task  for  system  health  was  to  identify 
and  locate  the  necessary  technical  data  for  all  system  tests  required  to  establish  that  it  is  fully 
operational. 

^  Helps  were  given  in  some  instances  when  requested  by  the  technician.  Helps  provided 
included  use  of  the  IMIS  or  paper  TO,  component  location,  interpretation  of  technical  data,  and 
use  of  the  multimeter.  Helps  were  intended  primarily  for  the  APG  technicians  to  compensate  for 
theii  lack  of  familiarity  with  the  test-bed  subsystems.  Helps  were  also  provided  to  the  specialists 
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Data  collection  forms  were  developed  for  each  test  problem.  Different  versions  were 
provided  for  the  problems  performed  with  the  paper  TO  and  for  problems  performed  with  the 
IMIS.  The  data  collection  forms  identify  the  steps  required  to  perform  the  task  in  the  order 
specified  by  the  paper  TO  or  IMIS.  Places  to  record  times  at  specified  check  points  and  ample 
space  were  provided  on  the  form  for  the  data  collector  to  record  times,  pertinent  information, 
and  observations.  In  addition,  a  supplemental  data  collection  sheet  was  provided  for  use  when 
the  technician  deviated  from  the  expected  path.  Sample  data  collection  sheets  are  provided  in 
Appendix  B. 


Other  Measures 

In  addition  to  the  performance  measures  listed  above,  three  other  measures  were  used  in 
this  effort:  the  National  Aeronautics  and  Space  Agency  Task  Load  Index  (NASA  TLX),  an 
IMIS  characteristics  questionnaire,  and  an  end  of  test  questionnaire 

a.  NASA  TLX.  The  NASA  TLX  provides  a  measure  of  the  work  load  imposed  by  a 
given  set  of  work  conditions.  After  completing  a  task  under  specified  working 
conditions,  the  subject  rates  the  workload  imposed  by  the  conditions  on  several 
factors  (mental  load,  physical  demands,  etc.).  The  ratings  are  then  combined  to 
provide  an  index  of  the  work  load.  Collecting  NASA  TLX  ratings  under  different 
working  conditions  provides  a  means  of  comparing  the  relative  workloads 
imposed  by  the  work  conditions  (NASA,  1986).  For  the  Field  Test,  the 
technicians  completed  the  NASA  TLX  after  each  test  problem.  The  NASA  TLX 
ratings  were  used  to  evaluate  the  relative  impact  of  the  IMIS  on  the  technicians 
perceived  workload.  The  NASA  TLX  was  administered  using  a  notebook 
computer. 

b.  IMIS  Characteristics  Questionnaire.  After  completion  of  all  assigned  tasks  using 
the  IMIS,  the  technicians  completed  a  questionnaire  designed  to  evaluate  various 
features  or  qualities  of  the  IMIS  (e.g.,  readability  of  the  display).  The  questions 
were  presented  in  the  Likert  format  using  a  notebook  computer.. 

c.  End  of  Test  Questionnaire.  After  completing  their  last  problem,  the  technicians 
completed  a  short  exit  questionnaire  which  asked  them  to  describe  what  they 
liked  and  disliked  about  the  IMIS,  and  what  should  be  done  to  make  it  better. 

Training 

Two  primary  types  of  training  were  provided  to  all  technicians  participating  in  the  study: 
(1)  how  to  use  the  IMIS,  and  (2)  how  to  use  the  paper  TOs  required  to  perform  the  test 
problems.  In  addition,  a  brief  lesson  was  given  on  the  use  of  the  multimeter  and  how  to  identify 


in  selected  situations.  These  helps  were  in  situations  where  the  technician  normally  would  have 
sought  help  from  another  technician  or  supervisor. 
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pins  on  multi-pin  connectors  (e.g.,  multi-pin  plugs/jacks).  This  training  was  given  the  day  before 
the  technicians  were  scheduled  to  begin  the  performance  tests. 

IMIS  Training 

The  IMIS  training  was  provided  to  develop  the  skills  needed  to  use  the  IMIS  to  perform 
the  fault  isolation  tasks  required  for  the  study.  Training  included  an  overview  of  the  IMIS 
system,  a  review  of  IMIS  capabilities,  log-on/off  procedures,  work  order  creation/close-out, 
parts  ordering  procedures,  accessing  job  assignments,  accessing  and  using  TOs,  and  using  the 
IMIS  diagnostic-aiding  capability.  The  training  included  a  practice  exercise  which  took  the 
technician  through  the  process  of  selecting  a  work  order  from  the  list  of  assigned  tasks, 
accessing  the  appropriate  fault  verification  procedures,  going  through  the  diagnostic  process, 
ordering  parts,  and  closing  out  the  work  order.  This  practice  exercise  demonstrated  all  the 
processes  required  to  perform  the  test  problems.  In  addition  to  the  formal  training,  the 
technicians  were  provided  an  opportunity  to  "play"  with  the  system  and  exercise  its  various 
features  on  their  own.  This  hands-on  experience  gave  them  an  opportunity  to  become 
comfortable  with  the  system.  This  training  required  approximately  four  to  five  hours. 

Paper  TO  Training 

Paper  TO  training  was  provided  to  ensure  the  technicians  were  fully  proficient  in  the  use 
of  the  TOs  required  to  perform  the  test  problems.  For  the  specialist  technicians,  this  training  was 
a  refresher  because  they  are  required  to  use  these  TOs  in  their  day-to-day  duties.  However,  the 
training  provided  new  material  to  the  APG  technicians  because  they  had  no  prior  training  or 
experience  with  the  avionics  fault  isolation  manuals.  This  training  included  a  review  of  the  TO 
numbering  system,  how  to  locate  and  use  job  guide  manuals,  fault  isolation  manuals,  wiring 
diagrams,  and  the  illustrated  parts  breakdown  TO.  An  emphasis  was  placed  on  how  the  different 
TOs  relate  to  each  other  and  how  to  "navigate"  through  the  TO  system  to  find  the  needed 
information.  As  part  of  the  training,  the  technician  completed  the  same  exercise  used  in  the-  - 
IMIS  training  to  illustrate  the  procedures  required  for  the  test.  This  training  required 
approximately  one  hour. 

Multimeter  Training 

Multimeter  training  was  provided  to  ensure  that  all  technicians  knew  how  to  properly  use 
the  multimeter  to  make  voltage  and  continuity  checks.  This  training  included  an  introduction  to 
the  Fluke  multimeter  and  instruction  on  making  voltage  and  ohms  readings  and  performing 
continuity  checks.  Emphasis  was  placed  on  avoiding  commonly  made  errors  while  conducting 
multimeter  readings  and  performing  checks.  In  addition,  training  was  given  on  how  to  identify 
pins  and  jacks  on  multi-pin  connectors.  The  various  schemes  used  to  identify  and  code  pins  on 
cannon  plugs  were  explained,  and  the  technicians  were  given  an  opportunity  to  practice  locating 
pins  on  typical  multi-pin  connectors. 
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IMIS/TO  Refresher  Training 


Following  the  completion  of  the  first  six  test  problems  using  either  IMIS  or  TOs,  the 
technicians  were  given  a  quick  refresher  on  the  type  of  data  to  be  used  for  the  next  block  of 
tests.  This  training  was  provided  to  ensure  the  technicians  were  proficient  in  the  use  of  the 
media  required  for  the  remaining  tasks. 


Technical  Data 


Paper  Technical  Orders 

The  standard  Air  Force  TOs  in  effect  at  the  time  of  the  test  were  used  in  the  data 
collection.  TOs  were  checked  out  from  the  Squadron  COSO. 

IMIS  Technical  Data 

Technical  data  for  the  three  subsystems  was  developed  for  the  IMIS  by  converting  the 
existing  TO  data  to  the  IMIS  format.  This  was  accomplished  by  converting  the  digital  data  base 
from  which  the  paper  TOs  were  printed  into  the  lETM  format  required  for  the  IMIS  (as 
specified  by  Military  Standard  MIL-D-87269).  The  conversion  process  left  the  technical  content 
the  same  but  added  the  codes  or  tags  required  to  present  the  data  on  a  computer  system.  The 
technical  content  of  the  IMIS  data  was  nearly  identical  to  the  TOs.  Some  questions  were 
restated  to  meet  electronic  presentation  requirements,  and  some  data  was  reformatted.  For 
example,  when  the  TO  referred  the  user  to  a  table  in  another  part  of  the  TO,  the  IMIS  presented 
the  relevant  data  rather  than  the  table.  The  IMIS  technical  data  was  developed  by  the  F-16 
prime  contractor,  Lockheed  Aircraft  Corporation,  Fort  Worth  Division. 

Test  and  Data  Collection  Procedures 

The  field  test  was  conducted  in  three  sessions.  The  first  two-week  session  was  a  pretest 
in  which  all  data  collection  procedures  were  tested,  refined,  and  practiced.  This  was  followed  by 
two  three- week  sessions  in  which  the  actual  testing  and  data  collection  was  accomplished. 
Approximately  one  week  separated  the  two  sessions.  Testing  and  data  collection  activities  were 
conducted  on  a  two-shift-per-day  basis,  0800  to  1700  and  1700  to  0100  hours.  Two  technicians 
were  tested  per  shift.  Approximately  four  days  were  required  to  test  each  pair  of  technicians,  in 
addition  to  one  day  for  training  and  orientation. 

Two  data  collection  teams,  one  per  shift,  administered  the  performance  tests  and 
collected  the  data.  Each  team  consisted  of  a  team  leader,  technical  observer,  and  data  collectors. 

a.  Team  Leader.  The  team  leader  is  responsible  for  technician  orientation  and 
training  as  well  as  scheduling.  Additionally,  the  team  leader  must  ensure  data 
collection  activities  are  conducted  according  to  the  established  rules  and  must 
resolve  any  problems  encountered. 
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b.  Technical  Observer.  The  technical  observer  is  responsible  for  installation  ofiaults 
for  test  problems;  close  (step-by-step)  monitoring  of  the  technician's  performance, 
determining  when  each  required  process  is  completed,  and  determining  when  the 
technician  has  successfully  completed  (or  failed)  the  task;  providing  guidance  to 
the  technician  on  test  procedures  to  be  followed;  and,  where  appropriate, 
answering  questions  and  providing  helps.  Additionally,  the  technical  observer 
keeps  the  data  collector  informed  of  progress.  This  is  accomplished  by  informing 
the  data  collector  when  the  technician  starts  or  completes  each  subtask,  when 
checkpoints  have  been  reached,  and  when  problems  are  encountered.  The 
technical  observers  in  this  effort  were  former  Air  Force  avionics  technicians  who 
were  thoroughly  familiar  with  F-16  maintenance  procedures,  the  test-bed 
subsystems,  and  associated  technical  data. 

c.  Data  Collector.  The  data  collector  is  responsible  for  monitoring  an  assigned 
technician's  performance,  recording  the  data  collected  for  each  problem  (including 
start/stop  times  at  each  checkpoint,  any  errors  made  by  the  technician,  and  any 
helps  given  by  the  technical  observer),  checking  close-out  information  entered  by 
the  technician  into  IMIS  or  the  work  order  form,  debriefing  the  technician  at  the 
end  of  each  problem,  and  ensuring  that  any  required  questionnaires  were 
completed.  Two  data  collectors  were  assigned  to  each  team.  Each  data  collector 
was  assigned  to  a  technician  and  monitored  that  technician's  performance 
throughout  the  test  period.  Data  collectors  were  selected  from  the  IMIS  project 
development  team.  The  personnel  assigned  were  thoroughly  familiar  with  the 
IMIS  and  maintenance  operations. 

The  data  collection  team  was  supported  by  a  software  specialist  who  prepared  PMA 
removable  disk  drives  and  resolved  any  hardware  or  software  problems  encountered. 

Data  Collection  Process 

Prior  to  the  start  of  the  test  session,  the  aircraft  was  prepared.  Preparations  included 
opening  all  panels  covering  areas  to  be  accessed  during  any  of  the  tests,  installing  the  breakout 
boxes  used  to  simulate  the  faults,  and  connecting  external  air  conditioning  and  power  to  the 
aircraft.  In  addition,  two  PMA  removable  disk  drives  (one  primary,  one  backup)  were  prepared 
for  each  problem  to  be  performed  that  day. 

Upon  arrival  the  first  day,  the  technicians  were  given  an  introduction  to  the  IMIS  project, 
the  goals  of  the  field  test  were  described,  and  the  technicians’  role  in  accomplishing  those  goals 
was  explained.  In  addition,  they  were  asked  to  complete  a  short  biographical  questionnaire  and 
to  sign  an  agreement  indicating  their  willingness  to  participate  in  the  study.  This  brief  orientation 
was  followed  by  training  on  the  IMIS,  TOs,  and  multimeter.  After  completing  the  training  and 
demonstrating  an  adequate  understanding  of  the  materials,  each  technician  was  assigned  to  one 
of  the  data  collectors  to  prepare  for  the  first  test.  In  some  cases,  data  collection  started  that 
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afternoon;  in  others  it  was  necessary  to  wait  until  the  next  day  (due  to  inadequate  time  or  the 
aircraft  being  in  use  by  another  test  subject). 

Technicians  completed  six  tasks  using  the  first  assigned  information  presentation  media 
(IMIS  or  paper  TO),  followed  by  a  refresher  on  the  use  of  the  other  media,  followed  by  the  six 
tasks  using  that  media.  Each  task  performance  was  followed  by  a  break  period.  Since  only  one 
technician  could  work  at  the  aircraft  at  one  time,  the  technicians  alternated  task  performance  and 
break  times,  so  that  when  one  technician  was  working  a  task,  the  other  technician  was  on  break. 
This  process  made  maximum  use  of  the  aircraft,  while  providing  ample  time  for  the  technicians 
to  take  breaks. 

Immediately  prior  to  the  first  task,  the  technicians  were  given  an  orientation  briefing  in 
which  the  data  collection  procedures  and  rules  were  explained.  The  briefing  emphasized  that 
the  technical  data  and  delivery  system  were  being  evaluated  not  the  technicians.  The  roles  of  the 
observer  and  data  collector  were  explained,  and  the  technicians  were  reminded  that  they  were 
free  to  ask  questions,  but  that  the  observer  may  or  may  not  answer  the  question  depending  on 
the  question  and  the  situation.  The  technician  was  instructed  to  always  use  the  IMIS  or  TO.^ 

After  the  briefing,  the  technicians  were  given  their  assigned  problem.  For  the  paper  TO 
condition,  the  technicians  were  given  a  paper  work  order  with  a  write-up  typical  of  those 
described  on  work  orders  created  during  the  pilot  debrief  and  were  told  to  work  the  problem. 
The  technicians  then  selected  the  required  TOs,  signed  them  out  from  the  "mini-TO  library"  in 
the  IMIS  office,  and  reported  to  the  aircraft.  For  the  IMIS  condition,  the  technicians  were  given 
a  PMA  (already  booted  and  ready  for  the  technician  to  sign-on)  and  were  told  that  the  assigned 
work  order  was  on  the  PMA.  Since  the  IMIS  PMA  contained  the  required  technical  data,  the 
technician  and  data  collector  proceeded  to  the  aircraft  to  start  the  problem.  The  technician 
started  the  task  by  accessing  the  work  order.  The  work  order  provided  the  information  as  it 
would  have  been  recorded  in  a  pilot  debrief  conducted  using  the  IMIS.  The  IMIS-based 
debriefing  process  prompts  the  debriefer  to  ask  questions  not  normally  asked  during  the  standard 
paper-based  debrief  These  questions  are  those  a  technician  assigned  to  work  the  write-up 
would  like  to  ask  the  pilot.  The  questions  help  to  define  the  problem  and  eliminate  possible 
causes,  thereby  speeding  up  the  diagnostic  process.  This  information  is  presented  on  the  IMIS 
work  order  for  the  technician  to  review  and  is  used  by  the  IMIS  in  developing  the  diagnostic 
process  to  be  followed. 

After  receiving  the  task  assignment,  the  technicians  reported  to  the  aircraft  to  start  the 
diagnostic  process.  Upon  arrival,  they  were  given  the  opportunity  to  ask  the  technical  observer 
(acting  as  the  pilot)  questions  on  system  performance  at  the  time  of  the  failure.  Although  the 
basic  steps  for  performing  the  tasks  (verify  fault,  isolate  cause  of  fault,  order  any  parts,  rectify 
fault,  verify  repair,  and  document  results)  are  the  same  for  the  IMIS  and  paper  TO  conditions. 


^  This  requirement  was  not  strictly  enforced.  Technicians  were  required  to  have  the  correct 
paper  TOs  available,  but  absolute  adherence  to  the  TOs  was  not  required.  Technicians  were 
allowed  to  proceed  without  interference  unless  they  were  about  to  make  a  serious  error  that 
could  cause  a  safety  hazard  or  damage  the  aircraft. 


21 


the  actions  required  to  accomplish  each  step  were  significantly  different.  The  processes  for 
accomplishing  the  steps  with  IMIS  and  paper  TOs  are  compared  in  Table  2.  The  data  collection 
process  is  illustrated  in  Figure  3. 

After  each  problem  the  technician  completed  the  NASA  TLX  ratings.  After  completing 
the  block  of  IMIS  tasks,  the  technicians  completed  the  IMIS  characteristics  questionnaire  which 
measured  their  reactions  to  using  the  IMIS  also.  After  all  testing,  the  technicians  completed  an 
end-of-test  questionnaire  giving  their  evaluation  of  IMIS  (what  they  liked  or  disliked)  and 
provided  any  suggestions  for  improving  the  IMIS. 

Data  Analysis 

Analysis  of  variance  was  used  to  analyze  the  data  collected  for  each  performance 
measure.  A  two  factor,  with  repeated  measures  on  one  factor,  design  was  used.  Between  group 
comparisons  were  made  for  Factor  A  (specialty  —  avionics  specialist  vs.  APG  technician)  and 
within  group  comparisons  for  Factor  B  (media  —  paper  TO  or  IMIS).  The  Scheffe  method  of 
comparisons  was  used  to  evaluate  differences  between  means  for  variables  identified  in  the 
analysis  of  variance  as  having  significant  differences.  All  computations  were  performed  using  the 
MANOVA  module  of  the  STATISTIC  A  software  package  (StatSoft,  1994). 


Table  2.  Comparison  of  Fault  Isolation  Process  Using  IMIS  vs.  Using  Standard 
_ Paper-Based  TOs  as  Implemented  for  Fault  Isolation  Test _ 


Action 

Current  Paper-Based  Procedures 

IMIS  Based  Procedures 

Preparation 

Identify  required  TOs,  gather  TOs,  sign  TOs 
out,  go  to  aircraft. 

Go  to  aircraft.  Connect  to  1553  interface. 

Verify  Fault 

Verify  fault  using  operator-initiated  BIT 
procedures. 

Verify  fault  with  MIS-controlled  BIT  (via 
1553  interface). 

Isolate  Fault 

Logic  tree  diagnostic  procedures. 

Dynamic  troubleshooting  procedures. 

Order  Part 

Return  to  COSO  (IMIS  Office),  look  up  part 
number  in  the  illustrated  parts  breakdown 
(IPB),  get  pro-super  approval  (data  collector 
acting  as  pro-super),  return  to  aircraft  and 
proceed  with  rectification. 

Select  "order  part"  from  dialog  box;  part 
order  forwarded  to  pro-super.  Proceed 
with  rectification. 

Rectify  Fault 

Locate  and  show  observer  proper  job  guide  for 
remove  and  replace  rectification.  Locate  and 
show  observer  wires  to  be  repaired  in  wiring 
diagram  and  connector  and  wiring  to  be 
repaired.  If  correct,  observer  declares 
rectification  complete.  If  incorrect,  observer 
tells  technician  repair  has  been  made  and  to 
continue.  Technician  uses  TO  to  identify 
required  system  health  checks.  Performs 
(actual)  system  health  to  verify  repair. 

System  health  fails;  fault  isolation  continued 
until  correct  solution  found. 

Select  "Replace  Component"  or  "Repair 
Wiring"  from  dialog  box.  PMA  presents 
required  instructions.  Show  to  observer. 

If  correct  rectification,  observer  declares 
rectification  complete.  Technician  presses 
continue  to  initiate  system  health.  If  not 
the  correct  rectification,  the  technician 
performs  system  health  using  1553 
interface.  System  health  fails;  fault 
isolation  continued  until  correct  solution 
found. 

System  Health 

Technician  uses  TO  to  determine  checks 
required  to  establish  system  health.  Shows 
observer  job  guides  required  to  perform 
system  health. 

Technician  continues  to  initiate  system 
health. 

Document 
Repair  Action 

Technician  returns  to  IMIS  office  and 
completes  simulated  Air  Force  Technical 

Order  (AFTO)  Form  349.  Accuracy  verified 
by  data  collector. 

Technician  uses  PMA  to  complete  close¬ 
out  form.  Accuracy  verified  by  data 
collector. 
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Limitations  of  the  Study 


Test  Environment  Limitations 

The  test  situation  imposed  the  following  limitations: 

a.  The  test  could  not  interfere  with  normal  maintenance  operations  of  the  host 
squadron  (310  FS). 

b.  No  activities  could  be  performed  which  risked  damage  to  the  aircraft  or  injury  to 
the  technicians  participating  in  the  study. 

c.  The  "live"  CAMS  data  base  could  not  be  used. 

d.  Time  constraints  required  some  short  cuts  which  added  a  degree  of  artificiality  to 
the  test  environment.  For  example,  aircraft  panels  were  opened  before  the  start 
of  the  test  and  standard  times  were  used  to  simulate  some  processes  (e.g., 
removal  and  replacement  of  parts). 

Demonstration  System  Limitations 

The  IMIS  system  used  for  the  study  was  built  as  a  proof-of-concept  demonstrator.  As  a 
result,  it  did  not  have  all  the  functionality  that  would  be  available  in  a  system  developed  for 
operational  use,  nor  did  it  fully  satisfy  the  performance  requirements  of  an  operational  system. 
Consequently,  performance  benefits  of  the  IMIS  are  not  fully  demonstrated  in  this  test.  The 
IMIS  Demonstration  System  had  the  following  limitations. 

a.  System  Performance.  The  times  required  for  the  system  to  process  information 
requests  and  present  technical  information  were  relatively  long. 

b.  System  Reliability.  The  system  was  prone  to  "freezing  up,"  requiring  a  delay  (i.e., 
time  out)  in  the  testing  process  while  the  problem  was  resolved. 


TEST  RESULTS 

The  test  results  are  presented  in  three  parts.  The  first  section  presents  the  results  of  the 
performance  tests.  The  second  section  presents  an  analysis  of  the  IMIS  diagnostics  capability  to 
reduce  parts  consumption.  Finally,  the  third  section  presents  the  results  of  the  NASA-TLX 
ratings,  questionnaire  data,  and  an  analysis  of  the  impact  of  the  IMIS  Demonstration  System 
speed  limitations. 
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Perfomiance  Test  Findings 


The  performance  test  results  are  presented  according  to  the  performance  measures 
discussed  in  the  previous  section  and  the  relevant  hypotheses.  The  results  for  successfiil  problem 
completion,  parts  used,  maintenance  errors,  and  the  overall  time  to  complete  each  problem  are 
presented  first,  followed  by  an  analysis  which  examines  various  components  of  the  observed 
times  to  determine  which  factors  contribute  most  to  the  differences  observed.  Finally,  an 
analysis  is  made  of  the  factors  contributing  to  observed  differences  in  parts  use.  Only  the  basic 
analysis  results  (means  and  levels  of  significance)  are  presented  in  this  section.  The  complete 
analysis  of  variance  tables  for  each  analysis  are  presented  in  Appendix  B. 

Performance  Measure:  Successful  Problem  Completion 

Hypotheses: 

1.  Avionics  specialist  technicians  will  successfully  complete  significantly  more 
diagnostic  problems  when  using  the  EMIS  as  the  source  of  technical  information 
than  when  using  the  paper  TO. 

2.  APG  technicians  using  the  IMIS  as  the  source  of  technical  information  will 
successfully  complete  significantly  more  diagnostic  problems  than  will  avionics 
specialists  using  paper  TOs  as  their  source  of  technical  information. 

Findings: 

Successful  completion  of  a  problem  was  defined  as  "satisfactorily  completing  all  subtasks 
required  to  return  the  aircraft  to  fully  operational  status."  Thus,  to  complete  the  problem,  the 
technician  was  required  to  verify  the  existence  of  the  reported  fault,  isolate  the  cause  of  the  feult, 
specify  the  required  rectification,  order  any  necessary  parts,  rectify  the  fault  (actual  repair 
simulated),  verify  that  the  system  was  fully  operational  (system  health  checks  simulated),  and 
complete  the  required  documentation.^  The  findings  for  this  variable  are  presented  in  Tables  3 
and  4.  Errors  in  completing  the  documentation  were  recorded,  but  not  considered  in 
determining  the  passing  or  failing  of  a  problem. 


^  Actual  rectifications  were  not  made.  The  technician  was  required  to  tell  the  technical  observer 
what  actions  he  would  take  to  make  the  rectification  and  show  the  specific  technical  data 
required.  The  system  health  checks  were  actually  made  on  the  aircraft,  if  the  rectification  just 
made  would  not  fix  the  problem.  If  the  correct  rectification  had  been  made,  the  system  health 
checks  were  simulated.  When  the  system  health  checks  were  simulated,  the  technician  was 
required  to  tell  the  observer  which  checks  he  would  make  and  to  show  the  instructions  for 
making  the  check  to  the  observer. 
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The  specialist  and  APG  technicians  successfully  completed  nearly  all  the  problems  when 
using  the  IMIS.  Only  one  problem  was  failed  when  using  the  IMIS,  compared  to  26  (of  144) 
problems  failed  using  the  paper  TO  as  the  source  of  technical  data. 


Table  3.  Percent  of  Problems  Successfully 
Completed  by  Avionics  Specialists 
and  APG  Technicians 


Subject  Category 

TO 

IMIS 

Avionics  Specialist 

81.94 

100.00 

APG  Technician 

69.44 

98.61 

Total 

75.69 

99.31 

Table  4.  Comparisons  of  Interest  for  Percentage  of  Problems  Successfully  Completed 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  IMIS 

75.69 

99.31 

-23.61 

Yes*** 

Spec.  TO  vs.  IMIS 

81.94 

100.00 

-18.06 

Yes** 

APG  TO  vs.  IMIS 

69.44 

98.61 

-29.17 

Yes*** 

Spec.  TO  vs.  APG  IMIS 

81.94 

98.61 

-16.67 

Yes* 

Spec.  TO  vs.  APG  TO 

81.94 

69.44 

12.50 

No 

Spec.  IMIS  vs.  APG  IMIS 

100.00 

98.61 

1.39 

No 

*  p  <  .05 

**p<.01 
***  p  <  .001 

The  success  rate  for  both  specialists  and  APG  technicians  was  much  lower  when  the 
paper  TO  was  the  source  of  technical  data.  Close  examination  of  the  data  reveals  that  most  of 
the  failures  with  the  paper  TO  were  due  to  a  failure  to  complete  all  required  system 
healthchecks.  These  differences  in  performance  can  be  explained  by  the  fact  one  or  more  BITs 
or  operational  checks  are  required  to  verify  that  the  system  has  been  returned  to  operationally 
ready  status.  System  health  tests  and  checkout  requirements  are  presented  in  the  follow-on 
maintenance  requirements  section  of  the  paper  TO.  The  manner  in  which  the  follow-on 
maintenance  requirements  are  presented  in  the  paper  TOs  for  some  systems  makes  it  easy  to 
overlook  required  checks.  Several  technicians  failed  to  complete  all  the  required  checks  and 
failed  the  problems  for  this  reason.  With  the  IMIS,  it  is  impossible  to  overlook  the  required 
checks.  When  a  technician  completes  a  task,  IMIS  immediately  presents  the  instructions  forthe 
follow-on  task.  He  either  has  to  follow  the  instructions  or  consciously  choose  not  to  do  the  task. 

Statistical  analysis  of  the  data  indicates  that  the  observed  differences  in  the  rate  of 
successful  problem  completion,  when  using  IMIS  vs.  paper  TO,  are  statistically  significant  for 
both  the  avionics  specialists  and  APG  technicians  (p  <.01  and  p  <.001,  respectively).  Further, 
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the  analysis  indicates  that  APG  technicians  using  the  IMIS  were  able  to  successfully  return 
significantly  more  systems  to  operational  status  than  specialists  using  paper  TOs  as  their  source 
of  technical  data.  This  difference  is  statistically  significant  (p  <.05).  Thus,  the  data  supports 
both  Hypothesis  1  and  Hypothesis  2.  Also,  it  should  be  noted  that,  on  this  measure,  the  APG 
technicians  using  the  IMIS  were  as  proficient  as  the  avionics  specialists  using  the  IMIS  (i.e.,  no 
significant  differences  were  found)  and  more  proficient  than  the  specialists  when  they  used  the 
paper  TO. 

Performance  Measure:  Parts  Used. 

Hypotheses: 

3.  Avionics  specialist  technicians  using  IMIS  to  perform  troubleshooting  tasks  will 
replace  significantly  fewer  parts  when  using  the  IMIS  than  when  using  paper  TOs. 

4.  APG  technicians  using  IMIS  to  perform  troubleshooting  tasks  will  replace 
significantly  fewer  parts  than  will  avionics  specialists  using  paper  TOs. 


Findings: 

Frequently  in  the  troubleshooting  process,  "good"  parts  are  unnecessarily  replaced. 

Good  parts  are  sometimes  replaced  as  part  of  the  diagnostic  strategy.  This  occurs  when  there  is 
no  way  to  determine  if  a  part  is  good  or  bad,  other  than  to  replace  it,  and  see  if  a  known  good 
part  fixes  the  problem.  Also,  good  parts  may  be  replaced  as  the  result  of  technician  error.  The 
IMIS  diagnostics  seek  to  reduce  the  number  of  good  parts  unnecessarily  replaced  by  providing  a 
strategy  which  reduces  the  frequency  of  instances  where  the  diagnostic  data  requires  replacing  a 
part  to  determine  if  it  is  good,  and  by  presenting  diagnostic  information  in  a  manner  which  will 
reduce  the  frequency  of  technician  errors  which  can  cause  a  good  part  to  be  replaced.  For  each 
set  of  six  problems  (two  FCR,  two  HUD,  two  INS)  used  in  this  test,  correctly  following  the 
diagnostic  procedures  in  the  paper  TOs  required  the  replacement  of  eight  parts  —  five  good 
parts  and  three  defective  parts.  The  corresponding  set  of  six  problems  performed  with  the  IMIS 
required  replacing  five  parts  —  two  good  parts  and  three  defective  parts.  The  good  parts  saved 
were  all  for  the  INS.  The  paper  TO  and  IMIS  required  the  same  number  of  parts  for  the  FCR 
and  HUD  problems.  The  INS  problems  required  replacing  five  parts  with  the  paper  TO  and  two 
parts  with  the  IMIS. 

The  mean  number  of  parts  used  by  technicians  to  complete  the  six  problems  under  each 
condition  are  shown  in  Tables  5  and  6.  The  specialists  required  an  average  of  8.67  parts  to 
complete  the  six  problems  using  the  paper  TO,  compared  to  6.42  parts  when  using  the  IMIS. 
The  APG  technicians  required  8.30  parts  for  the  problems  when  using  the  paper  TO,  compared 
to  5.30  parts  when  using  the  IMIS.  The  difference  in  the  mean  number  of  parts  used  with  IMIS 
vs.  the  paper  TO  were  statistically  significant  at  the  .001  level  of  confidence  for  both  the 
specialists  and  APG  technicians.  Thus,  Hypothesis  3  is  supported  by  the  data.  When  the  mean 
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number  of  parts  used  by  APG  technicians  is  compared  with  the  mean  for  specialists,  the 
difference  is  statistically  significant  (p  <  .001)  also.  Thus,  Hypothesis  4  is  supported  by  the  data. 


Table  5.  Mean  Number  of  Parts  Used  for  Six 


Problems  Under  Each  Condition 


Subject  Category 

TO 

IMIS 

Avionics  Specialist 

8.67 

6.42 

APG  Technician 

8.30 

5.30 

Total 

8.50 

5.91 

Table  6.  Comparisons  of  Interest  for  Mean  Number  of  Parts  Used  for  Six  Problems 


Under  Each  Condition 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  IMIS 

8.48 

5.90 

2.69 

Yes*** 

TO  vs.  IMIS 

8.67 

6.42 

2.25 

Yes** 

APG  TO  vs.  IMIS 

8.28 

5.30 

3.13 

Yes*** 

Spec.  TO  vs.  APG  IMIS 

8.67 

5.30 

3.51 

Yes*** 

Spec.  TO  vs.  APG  TO 

8.67 

8.28 

0.38 

No 

Spec.  IMIS  vs.  APG 

IMIS 

6.42 

5.30 

1.26 

No 

**  p<  .01 
***  p<  .001 


Analysis  of  the  data  revealed  that  both  the  specialists  and  APG  technicians  replaced 
significantly  more  good  parts  when  using  paper  TOs  than  when  using  IMIS  (p  <  .01  and 
p  <  .001,  respectively).  Thus,  Hypothesis  3  is  accepted.  In  addition,  specialists  using  the  paper 
TO  replaced  significantly  more  parts  than  APG  technicians  using  the  IMIS  (p  <  .001),  supporting 
Hypothesis  4.  Five  of  the  good  parts  replaced  when  using  the  paper  TO  were  directed  by  the 
paper  TO  as  part  of  its  troubleshooting  strategy.  The  additional  good  parts  (0.67  for  specialists 
and  0.30  for  APG  technicians)  replaced  when  using  the  paper  TO  were  due  to  technician  error. 
Two  of  the  good  parts  replaced  using  the  IMIS  were  directed  as  part  of  its  troubleshooting 
strategy.  The  additional  good  parts  (1.42  for  specialists  and  0.30  for  APG  technicians)  replaced 
when  using  the  IMIS  were  due  to  technician  error. 

Detailed  analysis  of  parts  usage  revealed  that  the  part  savings  for  the  IMIS  were  from 
one  subsystem,  the  INS.  This  appears  to  be  due  to  the  differences  in  the  complexity  of  the 
troubleshooting  tasks  for  the  systems.  The  INS  troubleshooting  procedures  are  much  more 
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complex  than  procedures  for  the  FCR  and  HUD.  A  detailed  analysis  of  the  ability  of  the  IMIS  to 
reduce  parts  requirements  is  presented  later  in  this  section. 


Performance  Measure:  Maintenance  Errors 

Hypotheses: 

5.  Avionics  specialist  technicians  will  make  significantly  fewer  maintenance  errors 
while  performing  the  troubleshooting  problems  when  using  the  IMIS  than  when 
using  paper  TOs. 

6.  APG  technicians  will  make  significantly  fewer  maintenance  errors  while 
performing  the  troubleshooting  problems  when  using  the  IMIS  than  will  avionics 
specialists  using  the  paper  TOs. 

Findings: 

As  indicated  above,  one  goal  of  the  IMIS  diagnostics  was  to  reduce  the  frequency  of 
errors  which  could  result  in  the  unnecessary  replacement  of  good  parts.  The  degree  to  which 
this  goal  was  met  by  the  IMIS  Demonstration  System  is  reflected  in  Tables  7  and  8. 

Table  7.  Mean  Number  of  Serious  Errors 


Under  Each  Condition 


Subject  Category 

TO 

IMIS 

Avionics  Specialist 

0.69 

0.29 

APG  Technician 

1.06 

0.18 

Total 

0.87 

0.23 

Table  8.  Comparisons  of  Interest  for  Mean  Number  of  Errors  per  Problem  Under 

Each  Condition 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  IMIS 

0.87 

0.23 

0.64 

Yes*** 

Spec.  TO  vs.  EMIS 

0.69 

0.29 

0.40 

No 

APG  TO  vs.  IMIS 

1.06 

0.18 

0.88 

Yes*** 

Spec.  TO  vs.  APG  IMIS 

0.69 

0.18 

0.51 

Yes* 

Spec.  TO  vs.  APG  TO 

0.69 

1.06 

-0.37 

No 

Spec.  IMIS  vs.  APG  EMIS 

0.29 

0.18 

0.11 

No 

*  p  <  .05 

***  p  <  .001 
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Maintenance  errors  made  while  performing  the  test  problems  were  recorded  by  the  data 
collector.*  Analysis  of  the  error  data  reveals  that  errors  made  by  avionics  specialists  were 
reduced  by  58  percent  and  errors  by  the  APG  technicians  were  reduced  by  83  percent.  The 
observed  difference  for  the  APG  technicians  is  statistically  significant  (p  <  .001).  However,  the 
observed  difference  for  the  specialists  was  not  statistically  significant.  Thus,  Hypothesis  5  is 
rejected.  Comparison  of  the  errors  made  by  specialists  using  the  paper  TO  with  the  number  of 
errors  made  by  the  APG  technicians  using  the  IMIS  revealed  that  the  APG  technicians  made 
significantly  fewer  errors,  supporting  Hypothesis  6.  The  IMIS  had  a  greater  impact  for  reducing 
errors  for  the  APG  technicians  than  for  the  specialists.  This  difference  is  reflected  in  a  significant 
statistical  interaction  (p  <  .05).  This  difference  in  impact  appears  to  be  due,  in  part,  to  their  lack 
of  familiarity  with  some  of  the  tasks  required  to  perform  the  problems  (e.g.,  taking  resistance 
readings  from  multi-pin  connectors). 

The  most  frequent  errors  for  the  paper  TO  condition  were  omitting  a  required  BIT  or 
operational  check,  use  of  the  wrong  operational  check,  misuse  of  the  paper  TO  (followed  wrong 
path,  used  wrong  procedure,  misread  data,  etc.),  and  errors  in  making  readings  with  the 
multimeter.  The  most  common  errors  for  the  IMIS  condition  were  errors  in  using  the  multimeter 
(read  wrong  pin,  read  from  jack  instead  of  plug,  etc.),  and  omission  of  a  required  step  (e.g., 
failure  to  reconnect  cannon  plug,  skipping  step,  etc.). 

Performance  Measure:  Total  Time 

Hypotheses: 

7.  Avionics  specialist  technicians  will  require  significantly  less  total  time  to  complete 
troubleshooting  problems  when  using  the  IMIS  than  when  using  paper  TOs. 

8.  APG  technicians  will  require  significantly  less  total  time  to  complete  the 
troubleshooting  problems  when  using  the  IMIS  than  will  avionics  specialists  using 
the  paper  TO. 

Findings: 

The  mean  times^  to  complete  the  troubleshooting  tasks  and  results  of  the  statistical 
analysis  are  shown  in  Tables  9  and  10. 


*  For  the  study,  an  error  was  defined  as  an  incorrect  action  which  could  result  in  a  failure  to 
diagnose  the  fault,  result  in  unnecessary  replacement  of  a  good  part,  significantly  increase  the 
troubleshooting  time  (by  sending  the  technician  down  the  wrong  path),  or  cause  an  injury  to  the 
technician  or  serious  damage  to  the  aircraft. 

^  The  times  in  Tables  9  and  10  are  adjusted  to  include  the  time  required  to  submit  and  process 
part  orders  and  the  time  required  to  input  close-out  data  into  CAMS.  This  adjustment  was  made 
by  adding  15  minutes  to  the  observed  time  for  each  part  ordered  and  10  minutes  to  the  observed 
time  to  adjust  for  the  time  to  enter  the  close-out  information  into  CAMS. 
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Table  9.  Mean  Problem  Performance 


Times  (in  Minutes)  for  Each  Problem 


Subject  Category 

TO 

MIS 

Avionics  Specialist 

149.29 

123.64 

APG  Technician 

175.82 

124.04 

Total 

161.46 

123.83 

Table  10.  Comparisons  of  Interest  for  Performance  Times 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  MIS 

161.46 

123.83 

37.63 

Yes*** 

Spec.  TO  vs.  MIS 

149.29 

123.64 

25.64 

Yes** 

APG  TO  vs.  MIS 

175.82 

124.04 

51.78 

Yes*** 

Spec.  TO  vs.  APG  MIS 

149.29 

124.04 

25.25 

Yes** 

Spec.  TO  vs.  APG  TO 

149.29 

175.82 

-26.53 

Yes** 

Spec.  MIS  vs.  APG  MIS 

123.64 

124.04 

-0.40 

No 

**  p<.01 
***  p  <  .001 


Both  the  avionics  specialists  and  APG  technicians  required  more  time  to  complete  the 
fault  isolation  problems  when  using  the  paper  TO.  Use  of  the  IMIS  reduced  problem 
performance  times  of  the  specialists  by  approximately  17  percent  and  the  times  of  the  APG 
technicians  by  approximately  29  percent.  The  performance  times  of  the  specialists  and  APG 
technicians  were  essentially  the  same  (i.e.,  no  statistically  significant  differences  were  found). 

The  statistical  analysis  indicates  that  when  using  the  IMIS,  the  specialists  were  able  to 
perform  the  troubleshooting  problems  in  significantly  less  time  than  when  using  the  paper  TO 
(p  <  .001),  supporting  Hypothesis  7.  In  addition,  the  APG  technicians  using  the  IMIS  were  able 
to  perform  the  problems  in  significantly  less  time  than  the  specialists  using  the  paper  TOs 
(p  <  .01).  Thus,  Hypothesis  8  is  supported,  also.  In  addition,  the  data  indicates  that  the  use  of 
the  IMIS  had  a  greater  relative  benefit  for  the  APG  technicians  than  the  specialists  (i.e.,  APG 
technicians  improved  more  with  IMIS).  The  IMIS  had  the  impact  of  improving  the  performance 
times  of  the  APG  technicians  from  being  much  slower  than  the  specialists,  to  being  on  a  par  with 
the  specialists.  This  finding  is  reflected  in  a  significant  statistical  interaction  (p  <  .001)  between 
the  media  (IMIS  vs.  paper  TO)  and  job  classification  (specialist  vs.  APG). 
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Performance  Measure;  Preparation  Time 

Hypotheses; 

9.  Avionics  specialist  technicians  will  require  significantly  less  time  to  prepare  for  a 
troubleshooting  task  when  using  the  IMIS  as  the  source  of  technical  information 
than  when  using  the  paper  TO. 

10.  APG  technicians  using  the  IMIS  will  require  significantly  less  time  to  prepare  for 
a  troubleshooting  task  than  will  avionics  specialists  using  paper  TOs. 

Findings; 

The  mean  preparation  times  and  results  of  the  statistical  analysis  of  the  observed 
preparation  times  are  presented  in  Tables  1 1  and  12.  As  may  be  observed  fi'om  the  tables,  only  a 
small  difference  in  preparation  times  was  observed  for  problems  performed  with  the  paper  TO 
versus  problems  performed  with  the  IMIS.  Also,  only  small  differences  in  preparation  times 
were  observed  for  specialists  and  APG  technicians.  Hypotheses  9  and  10  are  not  supported  by 
the  data.  Specialists  were  not  able  to  prepare  for  the  troubleshooting  task  significantly  faster 
when  using  IMIS  than  when  using  the  paper  TO,  and  APG  technicians  using  the  IMIS  were  not 
able  to  prepare  for  the  task  significantly  faster  than  the  specialists  using  the  paper  TO. 


Table  1 1 .  Mean  Preparation  Time  (in 


Minutes)  for  Each  Problem 


Subject  Category 

TO 

IMIS 

Avionics  Specialist 

5.00 

5.18 

APG  Technician 

5.10 

4.56 

Total 

5.05 

4.87 

Table  12.  Comparisons  of  Interest  for  Preparation  Times 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  IMIS 

5.05 

4.87 

0.18 

No 

Spec.  TO  vs.  IMIS 

5.00 

5.18 

-0.18 

No 

APG  TO  vs.  IMIS 

5.10 

4.56 

0.54 

No 

Spec.  TO  vs.  APG  IMIS 

5.00 

4.56 

0.44 

No 

Spec.  TO  vs.  APG  TO 

5.00 

5.10 

-0.10 

No 

Spec.  IMIS  vs.  APG  IMIS 

5.18 

4.56 

0.63 

No 
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Performance  Measure;  Fault  Verification  Time 


Hypotheses: 

1 1 .  Avionics  specialist  technicians  will  require  significantly  less  time  to  verify  faults  in 
the  test-bed  subsystems  when  using  the  IMIS  than  when  using  paper  TOs. 

12.  APG  technicians  using  IMIS  will  require  significantly  less  time  to  verify  faults  in 
the  test-bed  subsystems  than  will  avionics  specialists  using  paper  TOs. 

Findings; 

Tables  13  and  14  present  the  mean  fault  verification  times.  Examination  of  the  tables 
reveals  that  using  the  IMIS  to  verify  the  fault  did  not  have  a  significant  benefit  for  the  specialists. 
They  required  slightly  longer  to  verify  the  fault  when  the  IMIS  was  used.  This  difference  is  not 
statistically  significant;  thus.  Hypothesis  1 1  is  rejected.  However,  the  IMIS  did  make  a 
significant  difference  for  the  APG  technicians  when  verifying  the  fault.  Use  of  the  IMIS  reduced 
the  time  required  to  verify  the  fault  by  40  percent.  The  analysis  also  indicates  that  the  APG 
technicians  using  IMIS  were  not  able  to  verify  the  fault  significantly  faster  than  the  specialists 
using  the  paper  TO.  Thus,  Hypothesis  12  is  rejected  also. 


Table  13.  Mean  Fault  Verification  Times 


Subject  Category 

TO 

IMIS 

Avionics  Specialist 

9.65 

10.57 

APG  Technician 

18.31 

11.07 

Total 

13.98 

10.82 

Table  14.  Comparisons  of  Interest  for  Fault  Verification  Times 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  IMIS 

13.98 

10.82 

3.16 

Yes*** 

Spec.  TO  vs.  IMIS 

9.65 

10.57 

-0.92 

No 

APG  TO  vs.  IMIS 

18.31 

11.07 

7.24 

Yes*** 

Spec.  TO  vs.  APG  IMIS 

9.65 

11.07 

-1.42 

No 

Spec.  TO  vs.  APG  TO 

9.65 

18.31 

-8.65 

Yes*** 

Spec.  IMIS  vs.  APG  IMIS 

10.57 

11.07 

-0.50 

No 

***p  < .001 
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The  IMIS  had  a  much  greater  effect  for  the  APG  technicians  performing  the  fault 
verification  task  than  for  the  specialists.  Their  times  improved  from  being  significantly  slower 
than  the  specialist  times  to  being  approximately  equal.  This  difference  in  the  effect  of  the 
diagnostic-aiding  methodology  is  reflected  in  a  significant  statistical  interaction  (p  <  .001).  The 
interaction  shows  that  when  using  paper  TOs  there  was  a  significant  difference  between 
specialties;  however,  when  using  the  IMIS,  significant  differences  were  not  found. 

Performance  Measure:  Fault  Isolation  Time 

Hypotheses: 

13.  Avionics  specialist  technicians  will  require  significantly  less  time  to  isolate  faults 
in  the  test-bed  subsystems  when  using  the  IMIS  than  when  using  paper  TOs. 

14.  APG  technicians  using  IMIS  will  require  significantly  less  time  to  isolate  faults  in 
the  test-bed  subsystems  than  will  avionics  specialists  using  paper  TOs. 

Findings: 

The  mean  fault  isolation  times  are  given  in  Tables  15  and  16.  For  this  measure,  fault 
isolation  time  was  defined  as  the  performance  time  (starting  when  the  technical  observer  told  the 
technician  to  isolate  the  fault  [after  fault  verification]  and  ending  when  the  technician 


Table  15.  Mean  Fault  Isolation  Times  (in 
_ Minutes)  for  Each  Problem 


Subject  Category 

TO 

IMIS 

Avionics  Specialist 

116.22 

95.92 

APG  Technician 

132.72 

96.86 

Total 

123.79 

96.35 

Table  16.  Comparisons  of  Interest  for  Fault  Isolation  Times 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  IMIS 

123.79 

96.35 

27.44 

Yes*** 

Spec.  TO  vs.  IMIS 

116.22 

95.92 

20.31 

Yes** 

APG  TO  vs.  IMIS 

132.72 

96.86 

35.86 

Yes*** 

Spec.  TO  vs.  APG  IMIS 

116.22 

96.86 

19.36 

Yes** 

Spec.  TO  vs.  APG  TO 

116.22 

132.72 

-16.50 

No 

Spec.  IMIS  vs.  APG 

95.92 

96.86 

-0.94 

No 

IMIS 

**p<.01 

***  < .001 
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declared  the  rectification  complete  and  the  subsystem  returned  to  full  operational  status)  in 
addition  to  standard  times  for  rectifications  made,  part  ordering,  and  the  final  system  health. 

Both  the  specialists  and  APG  technicians  were  able  to  isolate  the  faults  significantly  fester 
(p  <  .05  and  p  <  .001  respectively)  when  using  the  IMIS  than  when  using  the  paper  TO.  Thus, 
Hypothesis  13  is  supported.  In  addition,  the  APG  technicians  using  the  IMIS  required 
significantly  less  time  than  the  specialists  to  isolate  the  faults  (p  <  .05),  thereby  supporting 
Hypothesis  14.  Although  there  was  a  relatively  large  (but  not  statistically  significant)  difference 
between  the  times  for  specialists  and  APG  technicians  when  both  used  the  paper  TOs,  the  use  of 
the  IMIS  brought  the  performance  of  the  APG  technicians  to  a  level  approximately  equal  to  the 
specialists.  Thus,  the  IMIS  had  a  greater  benefit  for  the  APG  technicians.  This  observation  is 
reflected  in  a  significant  statistical  interaction  (p  <  .001)  for  the  specialty  and  media  (IMIS  vs. 
paper  TO)  variables. 

Performance  Measure;  Part  Ordering  Time 

Hypotheses; 

15.  Avionics  specialist  technicians  will  require  significantly  less  time  to  order  each 
part  when  using  the  IMIS  than  when  using  current  part  ordering  procedures. 

1 6.  APG  technicians  using  IMIS  will  require  significantly  less  time  to  order  each  part 
than  required  by  avionics  specialists  using  current  part  ordering  procedures. 

Findings; 

The  mean  times  to  order  each  part  are  shown  in  Tables  17  and  18.  Since  there  was  no 
way  to  simulate  the  part  ordering  process  for  the  current  procedures  without  interfering  with 
squadron  operations,  a  standard  time  estimate  was  used  for  this  measure.  The  estimate  used  for 
each  part  order  was  the  observed  time  for  the  technician  to  locate  the  necessary  part  ordering 
information  (part  number,  figure  number,  etc.)  plus  a  standard  time  for  traveling  to  the  COSO, 
getting  supervisor  approval,  and  processing  the  order.  A  standard  time  of  15  minutes  was  used 
for  each  part  order.  The  estimate  was  based  on  estimates  by  experienced  technicians, 
supervisors,  and  COSO  personnel,  and  is  considered  to  be  conservative.  Analysis  of  the  data 
indicates  that  both  the  specialists  and  APG  technicians  were  able  to  order  parts  significantly 
faster  (p  <  .001)  when  using  the  IMIS  to  order  parts  than  when  they  use  the  current  part 


Table  17.  Mean  Part  Ordering  Time 


Subject  Category 

Current 

IMIS 

Avionics  Specialist 

19.42 

1.16 

APG  Technician 

25.28 

1.47 

Total 

22.29 

1.30 
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Table  18.  Comparisons  of  Interest  for  Mean  Part  Ordering  Time 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  Current  vs.  IMIS 

22.29 

1.30 

-5.86 

Spec.  Current  vs.  IMIS 

19.42 

1.16 

18.26 

Yes*** 

APG  Current  vs.  IMIS 

25.28 

1.47 

23.81 

Yes*** 

Spec.  Current  vs.  APG  IMIS 

19.42 

1.47 

17.95 

Spec.  Current  vs.  APG  TO 

19.42 

25.28 

-5.86 

Yes*** 

Spec.  IMIS  vs.  APG  IMIS 

1.16 

1.47 

-0.31 

No 

***  p<  .001 


ordering  process.  In  addition,  the  analysis  indicates  that  the  APG  technicians  using  the  IMIS 
required  significantly  less  time  to  order  parts  (p  <  .001)  than  did  specialists  using  the  paper  TO. 
Consequently,  Hypotheses  15  and  16  are  both  supported  by  the  data.  The  analysis  also  revealed 
a  significant  statistical  interaction  (p  <  .001)  between  specialty  and  media,  indicating  that  the 
IMIS  had  more  effect  on  part  ordering  time  for  the  APG  technicians  than  for  the  specialists. 

The  greatly  reduced  part  ordering  times  when  the  IMIS  is  used  are  primarily  due  to  the 
use  of  the  RF  link.  The  IMIS  transmits  the  part  order  request  directly  to  the  Production 
Superintendent  for  approval.  After  approval,  the  order  is  directly  transmitted  to  the  CAMS  and 
SBSS  to  be  filled.  The  current  procedure  requires  the  technician  to  go  to  the  COSO,  look  up  the 
part  number,  fill  out  the  part  order,  locate  the  Production  Superintendent,  get  his  or  her 
approval,  and  take  the  order  to  the  supply  clerk  for  entry  to  the  SBSS. 

Performance  Measure:  Close-Out  Time 

Hypotheses: 

17.  Avionics  specialist  technicians  will  require  significantly  less  time  to  complete 
required  documentation  when  using  the  IMIS  than  when  using  current 
documentation  and  close-out  procedures. 

18.  APG  technicians  using  IMIS  will  require  significantly  less  time  to  complete 
required  documentation  than  will  avionics  specialists  using  current  documentation 
and  close-out  procedures. 

Findings: 

Test  constraints  did  not  permit  a  direct  comparison  of  the  current  close-out  procedures 
and  the  IMIS-based  close-out  procedures  because  the  current  procedures  require  the  technician 
to  locate  a  CAMS  terminal  and  enter  the  data  to  the  CAMS  system.  Since  there  was  no  way  to 
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do  this  in  a  test  situation  without  inputting  the  test  data  into  the  CAMS  (and  thus  contaminating 
the  CAMS  data  base),  it  was  necessary  to  simulate  the  CAMS  input  for  the  current  process 
condition.  This  simulation  was  accomplished  by  having  the  technicians  enter  the  necessary 
information  on  a  paper  form.  The  time  required  to  complete  the  form  was  recorded  and  a 
standard  time  for  entering  the  data  to  the  CAMS  was  added  to  the  observed  time  to  provide  an 
estimated  close-out  time.  The  standard  time  was  developed  from  estimates  provided  by 
experienced  technicians.  The  technicians  estimated  that  the  process  of  going  to  the  maintenance 
office,  obtaining  access  to  a  CAMS  terminal,  and  entering  the  data  requires  approximately  15 
minutes.  A  conservative  estimate  of  10  minutes  per  problem  was  used  as  the  standard  time  for 
this  measure. 

The  work  order  close-out  procedure  implemented  in  the  IMIS  Demonstration  System 
required  the  technician  to  enter  the  required  information  on  a  form  displayed  on  the  screen.  To 
complete  the  form,  the  technician  selected  the  desired  entries  from  lists  of  possible  entries.  The 
completed  information  was  then  transmitted  via  the  RF  link  to  the  CAMS.  This  process  saved 
the  time  required  for  the  technician  to  locate  a  CAMS  terminal  and  enter  the  data.  The  IMIS 
work  order  close-out  was  a  slow  process,  given  the  slow  response  times  of  the  IMIS 
Demonstration  System.  It  also  provided  an  opportunity  for  error, 

The  mean  close-out  times  are  given  in  Tables  19  and  20.  The  analysis  indicates  that  the 
IMIS  significantly  reduces  the  time  required  to  complete  close-out  documentation.  The  total 
close-out  times  are  reduced  by  approximately  46  percent.  All  comparisons  of  paper  TO-based 
close-out  with  IMIS-based  close-out  are  statistically  significant  (p  <  .001).  Specialists  were  able 
to  complete  the  close-out  significantly  faster  when  using  the  IMIS  than  when  using  paper  TOs. 
Thus,  Hypothesis  17  is  supported.  Also,  APG  technicians  using  the  IMIS  were  able  to  perform 
the  close-out  task  significantly  faster  than  specialists  using  paper  TOs.  Therefore,  Hypothesis  18 
is  accepted. 


Table  19.  Mean  Close-Out  Times 
_ (in  Minutes) _ 


Subject  Category 

TO 

IMIS 

Avionics  Specialist 

14.67 

8.17 

APG  Technician 

17.31 

8.82 

Total 

15.98 

8.49 

The  BVUS  Demonstration  System  did  not  fully  implement  the  IMIS  requirements  for 
documenting  maintenance  actions.  Most  of  the  information  required  for  completion  of  the  work 
order  close-out  (work  unit  code,  part  numbers,  etc.)  can  be  automatically  collected  during  the 
diagnostic  session  and  automatically  inserted  into  the  close-out  record  at  the  end  of  the  session. 
The  technician  would  not  be  required  to  do  more  than  verify  the  information  before  it  is 
submitted.  The  IMIS  Demonstration  System  did  not  automatically  insert  this  information.  An 
operational  implementation  of  the  IMIS  will  provide  this  automated  data  collection  and 
recording  function. 
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Table  20.  Comparisons  of  Interest  for  Close-Out  Time 


Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  IMIS 

15.98 

8.49 

7.49 

Yes*** 

Spec.  TO  vs.  IMIS 

14.67 

8.17 

6.50 

Yes*** 

APG  TO  vs.  IMIS 

17.31 

8.82 

8.49 

Yes*** 

Spec.  TO  vs.  APG  IMIS 

14.67 

8.82 

5.85 

Spec.  TO  vs.  APG  TO 

14.67 

17.31 

-2.64 

Yes** 

Spec.  IMIS  vs.  APG  IMIS 

8.17 

8.82 

-0.65 

No 

**  p  <  .01 
***  p  <  .001 


Performance  Measure;  Documentation  Errors 

Hypotheses 

19.  Avionics  specialist  technicians  will  make  significantly  fewer  documentation  errors 
when  using  the  IMIS  than  when  using  current  documentation  procedures. 

20.  APG  technicians  using  IMIS  will  make  significantly  fewer  documentation  errors 
than  will  avionics  specialist  technicians  using  current  documentation  procedures. 

Findings; 

Documentation  errors  are  summarized  in  Tables  21  and  22.  Examination  of  the  tables 
reveals  that  relatively  few  errors  were  made  per  problem.  No  significant  differences  were  found 
in  the  number  of  documentation  errors  made  by  specialists  and  APG  technicians.  In  addition, 
there  were  no  significant  differences  in  the  number  of  errors  made  when  using  the  IMIS  vs. 
using  current  documentation  procedures.  However,  there  was  a  significant  statistical  interaction 
(p  <  .05)  for  media  vs.  specialty.  The  interaction  indicates  that  the  IMIS  was  more  successful  in 
reducing  documentation  errors  for  APG  technicians  than  for  specialists.  In  the  paper  TO 
condition,  APG  technicians  made  more  documentation  errors  than  the  specialists.  In  the  IMIS 
condition,  specialists  made  more  documentation  errors  than  the  APG  technicians.  It  should  be 
noted  that  the  number  of  documentation  errors  made  was  relatively  small  and  not  of  much 
operational  significance.  As  noted  earlier,  a  full  implementation  of  the  IMIS  concept  will 
automatically  complete  the  documentation,  eliminating  almost  all  errors. 

Relative  Impact  of  IMIS  for  Specialists  and  APG  Technicians 

When  the  mean  times  for  the  various  measures  are  examined  as  a  whole,  a  major  benefit 
of  the  IMIS  is  obvious.  The  use  of  IMIS  essentially  "evens  out  the  playing  field"  for  specialists 
and  APG  technicians.  When  the  APG  technician  uses  the  IMIS,  the  performance  advantages  of 
the  specialist  disappear.  This  effect  is  clearly  illustrated  in  Figure  4.  When  the  APG  technicians 
used  the  IMIS,  their  performance  was  essentially  equal  to  the  specialists  using  the  IMIS  and 
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Table  2 1 .  Mean  Number  of  Documentation 
_ Errors  per  Problem _ 


Subject  Category 

TO 

IMIS 

Avionics  Specialist 

0.01 

0.11 

APG  Technician 

0.08 

0.04 

Total 

0.05 

0.08 

Table  22.  Comparisons  of  Interest  for  Documentation  Errors 

Comparison 

Mean 

Mean 

Difference 

Significant 

Total  TO  vs.  IMIS 

0.05 

0.08 

-0.03 

No 

Spec.  TO  vs.  IMIS 

0.01 

0.11 

-0.10 

No 

APG  TO  vs.  IMIS 

0.08 

0.04 

0.04 

No 

Spec.  TO  vs.  APG  IMIS 

0.01 

0.04 

-0.03 

No 

Spec.  TO  vs.  APG  TO 

0.01 

0.08 

-0.07 

No 

Spec.  IMIS  vs.  APG 

IMIS 

0.11 

0.04 

0.07 

No 

better  than  the  specialists  using  the  paper  TO.  With  the  exception  of  fault  verification,  there 
were  no  significant  differences  in  the  mean  times  observed  for  any  of  the  measures  used  in  this 
test  when  both  the  specialists  and  APG  technicians  used  the  IMIS.  However,  with  the  exception 
of  preparation  times,  there  was  a  significant  difference  for  all  measures  in  the  performance  of 
specialists  and  APG  technicians  when  both  used  the  paper  TO.  The  graphs  in  Figure  4  also 
illustrate  the  statistically  significant  interactions  observed  for  several  of  the  measures. 

Statistically  significant  interactions  were  found  for  total  time  (p  <  .01),  fault  verification 
(p  <  .001),  close-out  time  (p  <  .05),  maintenance  errors  (p  <  .05),  part  order  time  (p  <  .001),  and 
documentation  errors  (p  <  .05).  The  interaction  indicates  that  the  benefits  of  the  IMIS  are 
greater  for  the  APG  technicians  than  for  the  specialists. 

Analysis  of  IMIS  Diagnostics  Capability 

The  IMIS  Diagnostics  Module  is  expected  to  produce  significant  reductions  in  the 
consumption  of  spare  parts,  therefore  it  is  essential  that  these  capabilities  be  fiilly  understood. 

To  this  end,  a  thorough  analysis  was  made  of  the  IMIS  diagnostic  capability  to  minimize 
unnecessary  part  replacements.  The  analysis  was  conducted  in  three  parts:  a  more  thorough 
examination  of  the  field  test  parts  consumption  data,  a  table-top  comparison  of  the  diagnostic 
strategies  developed  by  IMIS  versus  diagnostic  strategies  used  in  the  paper  TO,  and  a 
comparison  of  projected  parts  usage  based  upon  subsystem  failure  data.  These  analyses  are 
based  upon  the  FCR,  HUD,  and  INS  paper  TOs,  and  the  diagnostic  strategies  developed  by  the 
IMIS  firom  the  EVES  data  base  for  those  subsystems.  The  results  of  the  analyses  are  presented:  in 
the  following  paragraphs. 
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Problem  Completion 


Problem  Performance  Time 


Close-Out  Times 


Major  Maintenance  Errors 


Parts  Used 


■  Specialist 

APG  Technician 


Part  Ordering  Time 


Figure  4.  Plots  of  Means  Demonstrating  Impact  of  IMIS  for  Avionics  Specialists 

and  APG  Technicians 
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Further  Analysis  of  Field  Test  Parts  Data 


Parts  use  by  the  technicians  for  all  problems  are  summarized  in  Table  23.  The  table 
presents  information  on  the  following  elements. 

a.  Expected  Part  Use.  The  “expected  part  use”  indicates  the  total  number  of  parts 
all  subjects  were  expected  to  use  for  all  problems  under  each  condition  when  the 
IMIS  or  paper  TO  are  followed  without  deviation  or  error, 

b.  Actual  Part  Use.  The  “actual  part  use”  represents  the  number  of  parts  actually 
used  by  all  subjects  under  each  condition. 

c.  Unnecessary  Removals.  “Unnecessary  removals”  are  the  number  of  parts  used  in 
excess  of  the  number  of  parts  required  to  complete  the  test  problems  by  following 
the  diagnostic  procedures  without  deviation  or  error  (actual  part  use  minus 
expected  part  use). 

d.  Parts  Not  Used.  The  “parts  not  used”  indicates  the  number  of  parts  "saved"  by 
deviating  from  the  diagnostic  procedure  (due  to  technicians  choice  based  upon 
subsystem  knowledge  or  a  lucky  error  which  led  to  a  quicker  solution  of  the 
problem). 

Table  23  reveals  that  the  technicians  replaced  many  more  parts  than  required  to  actually  fix  the 
problems.  A  total  of  72  parts  were  required  under  each  condition  to  solve  the  problems;  yet,  202 
parts  were  used  with  the  paper  TO  and  138  were  used  with  the  IMIS.  Thus,  130  good  parts 
were  replaced  with  the  paper  TO  and  66  good  parts  were  replaced  with  the  IMIS.  The  paper 
TO  directed  1 18  of  the  good  part  removals  as  part  of  the  diagnostic  strategy.  The  IMIS  led  to 
replacing  46  parts  as  part  of  the  diagnostic  strategy.  An  additional  12  good  parts  were  replaced 
with  the  paper  TO  due  to  technician  error  or  a  conscious  decision  to  deviate  from  the  paper  TO 
procedure.  Twenty  good  parts  were  replaced  with  the  IMIS  because  of  technician  error  or 
intentional  deviation.  In  addition,  there  were  three  instances  where  the  technician  used  one  less 
part  than  required  by  correctly  following  the  diagnostic  procedures.  These  instances  all  occurred 
when  the  paper  TO  was  used. 

The  data  clearly  illustrate  one  of  the  major  problems  encountered  in  avionics  diagnostics 
—  ambiguity  groups.  An  ambiguity  group  is  two  or  more  components  in  a  system  which  could 
be  the  cause  of  the  fault  under  consideration  for  which  there  is  no  way  to  eliminate  a  component 
from  consideration  without  replacing  it  or  another  component.  The  IMIS  was  successful 


^  1  There  were  two  instances  in  which  the  technician  was  not  able  to  correctly  identify  the  faulty 
part.  Data  for  these  problems  is  not  included  in  this  analysis. 
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Table  23.  Summary  of  Parts  Used  by  All  Technicians  for  All  Problems 


Specialist 

FCR 

HUD 

INS 

Total 

Specialist 

APG 

FCR 

HUD 

INS 


Expected  Part  Use 

Actual  Part  Use 

Unnecessary 

Parts 

Parts  Not  Used 

TO  IMIS 

TO  IMIS 

TO  IMIS 

TO  IMIS 

Total  APG  94 


98  61 


Total  All  190 
Subjects 


118  202  138  12  20 


in  breaking  the  ambiguity  groups  for  two  of  the  INS  problems  (by  recommending  a  quick  test 
before  making  a  rectification),  resulting  in  saving  three  good  parts  per  problem.  However,  the 
IMIS  was  not  able  to  break  the  ambiguity  groups  in  the  other  problems  and  used  the  same 
number  of  parts  as  the  paper  TO.  Dealing  with  ambiguity  groups  is  a  complex  issue  which  is 
examined  in  more  detail  in  the  discussion  section  of  this  report. 

For  several  problems,  technicians  replaced  more  than  the  expected  number  of  parts  for 
both  the  IMIS  and  paper  TO  conditions.  These  additional  parts  were  replaced  either  due  to  an 
error  made  by  the  technician  or  due  to  a  decision  by  the  technician  to  deviate  from  the  prescribed 
procedure.  Those  instances  where  extra  parts  were  used  were  examined  to  identify  the  reasons 
for  the  use  of  extra  parts.  The  apparent  reasons  for  each  use  of  an  extra  part  are  summarized  in 
Table  24. 


Table  24  presents  several  items  worth  noting.  The  major  cause  of  unnecessary  part  use 
for  both  the  paper  TO  and  IMIS  was  errors  made  in  taking  multimeter  readings.  The  errors  were 
due  to  the  technician  taking  measurements  from  the  wrong  pin  or  from  the  jack  instead  of  the 
plug  (or  vice  versa).  This  type  of  error  appears  to  occur  because  of  inadequate  guidance  in  the 
technical  data  (both  paper  TO  and  IMIS)  and  because  it  is  easy  to  miscount  pins  on  the  plug  or 
jack.  Earlier  test  versions  of  IMIS  diagnostics  have  provided  improved  illustrations  to  help 
technicians  locate  the  correct  pins.  Results  from  those  tests  suggest  that  the  use  of  similar 
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Table  24.  Reasons  for  Unnecessary  (Unexpected) 
Parts  Use 


Paper  TO  Problems 

Number 

Errors 

Number 

Parts* 

Use  of  Technical  Data 

5 

6 

Incorrect  Measurements 

4 

6 

Technicians  Choice 

1 

2 

IMIS  Problems 

Use  of  Technical  Data 

3 

3 

Incorrect  Measurements 

6 

11 

Tech.  Choice  (best  actions  list) 

5 

6 

*  Some  errors  led  to  replacing  more  than  one  part  unnecessarily 
(i.e.,  continuing  down  the  wrong  path). 

illustrations  in  the  IMIS  technical  data  may  have  reduced  the  number  of  errors  and  unnecessary 

component  replacements. 

Parts  were  unnecessarily  replaced  as  a  result  of  incorrect  use  of  both  the  paper  TO  and 
the  IMIS.  Errors  made  using  the  paper  TO  were  due  to  misreading  or  misinterpreting  the  data, 
incorrectly  following  the  procedure  (e.g.,  taking  the  wrong  path  in  a  troubleshooting  tree  or 
using  the  wrong  table),  or  the  technician  losing  his  place  (e.g.,  wind  turning  the  page  withoutthe 
technician  noticing).  The  errors  with  the  IMIS  were  due  to  the  technician  not  correctly 
following  the  procedures  or  misinterpreting  them. 

Several  parts  were  replaced  because  the  technician  consciously  chose  to  replace  a  part 
instead  of  following  the  procedure.  This  occurred  when  the  technician  thought  he  knew  the 
cause  of  the  fault  and  chose  to  take  a  short  cut.  Proper  use  of  the  paper  TO  (as  required  in  this 
study)  does  not  permit  deviation  from  the  technical  data.  However,  some  technicians  chose  to 
deviate  and  replace  a  part  based  upon  their  system  knowledge.  One  of  those  instances  resulted 
in  the  unnecessary  replacement  of  two  parts.  There  were  three  instances  when  the  technicians 
deviated  from  the  recommended  sequence,  one  less  part  was  used  than  would  have  been  required 
by  adhering  to  the  data. 

The  IMIS  permits  technicians  to  take  advantage  of  their  system  knowledge  and  deviate 
from  the  step-by-step  diagnostic  procedures.  This  option  is  provided  by  the  best  actions  list 
which  the  technician  can  call  up  during  the  diagnostic  process.  The  list  gives  ranked  lists  of  the 
tests  and  rectifications  available  at  that  time  and  the  probability  of  the  test  passing  or  the  listed 
rectification  correcting  the  problem.  The  first  item  in  the  list  is  always  the  recommended  action; 
however,  the  technician  can  choose  other  lower-ranked  options.  When  the  technician  selects 
from  the  list,  the  system  presents  the  technical  data  required  to  accomplish  that  action.  Several 
technicians  chose  to  take  advantage  of  the  best  actions  option  (i.e.,  they  did  not  choose  the 


^2  The  illustrations  used  in  the  IMIS  were  taken  from  the  paper  TO  with  little  modification  other 
than  to  remove  some  unnecessary  detail  and  clutter. 
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recommended  action)  and  used  it  to  replace  parts  at  their  discretion.  In  five  instances  where  the 
technician  chose  to  use  the  best  actions  option,  six  extra  parts  were  used.  There  were  no 
instances  where  use  of  the  best  actions  option  resulted  in  using  fewer  than  the  expected  number 
of  parts. 

Four  of  the  unnecessary  part  replacements  with  the  paper  TO  and  six  of  the  replacements 
with  the  IMIS  can  be  classified  as  continuations.  These  were  parts  replaced  as  the  result  of  an 
error  that  had  already  led  to  the  replacement  of  one  or  more  good  parts.  This  type  of  part  use 
occurred  when  the  technician  made  an  error  which  led  him  down  the  wrong  path,  or  made  a  part 
replacement  based  upon  his  system  knowledge  which  led  to  further  replacements  when  the  fault 
was  not  corrected. 

The  data  presented  in  Table  23  also  provides  an  explanation  for  the  finding  that  the  parts 
savings  were  all  for  the  INS.  Comparison  of  the  expected  parts  use  for  the  paper  TO  versus  the 
expected  parts  use  for  the  IMIS  reveals  that  error-fi'ee  performance  of  the  FCR  and  HUD  tasks 
using  the  paper  TO  would  require  the  same  number  of  parts  as  error-free  performance  using  the 
IMIS.  However,  error-free  performance  using  the  paper  TO  for  the  two  INS  problems  requires 
five  parts  per  subject,  while  error-free  performance  using  the  IMIS  requires  only  two  parts  per 
subject.  It  was  expected  that  the  use  of  the  IMIS  diagnostics  also  would  have  required  fewer 
parts  for  the  FCR  and  HUD  problems.  There  are  two  explanations:  (1)  the  evaluation  team 
chose  FCR  and  HUD  faults  that  IMIS  cannot  improve  upon  the  existing  paper  TOs  (the  paper 
TO  is  as  efficient  as  possible  given  the  available  subsystem  performance  information  for  that 
fault),  and  (2)  the  IMIS  diagnostics  do  not  yield  the  expected  benefits  for  all  subsystems. 

Examination  of  the  FCR  and  HUD  fault  trees  used  in  the  test  suggest  that  the  problem 
was  basically  the  "luck  of  the  draw."  Other  faults  could  have  been  selected  from  the  trees  as  the 
test  faults  which  would  have  required  fewer  parts  when  using  IMIS  diagnostics  than  when  using 
the  paper  TO  diagnostics.  Although,  this  finding  explains  why  better  results  were  not  achieved 
for  the  test  faults,  it  does  raise  the  question  of  how  often  the  IMIS  diagnostics  require  fewer 
parts.  In  other  words,  what  is  the  real  potential  of  the  IMIS  diagnostics  for  reducing  parts 
consumption  on  the  F-16  INS,  FCR,  and  HUD?  An  additional  analysis  was  made  to  answer  this 
question.  The  results  are  provided  in  the  following  sections. 

Comparison  of  Paper  TO  and  IMIS  Fault  Trees 

The  12  faults  used  as  the  basis  for  the  problems  for  the  IMIS  field  test  represent  only  a 
small  portion  of  the  possible  faults  in  the  three  test-bed  subsystems.  Although,  they  were 
selected  to  be  representative  of  the  faults  occurring  in  the  subsystems,  it  is  not  known  how 
representative  the  paper  TO  fault  trees  used  actually  are.  Nor,  is  it  known  if  the  diagnostic 
procedures  generated  by  the  IMIS  for  the  12  faults  are  representative  of  the  fault  trees  that 
would  be  generated  to  isolate  all  faults  in  the  subsystem.  To  answer  this  question  and  to  obtain  a 
better  understanding  of  the  relative  efficiency  of  the  IMIS  and  paper  TO  fault  trees,  a  table-top 
analysis  was  accomplished  to  compare  the  efficiency  (in  terms  of  part  use)  of  the  paper  TO  fault 
trees  with  the  fault  trees  generated  by  the  IMIS. 
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The  analysis  was  accomplished  by  selecting  a  representative  sample  of  fault  trees  for  each 
of  the  three  subsystems.  Eight  fault  trees  were  selected  for  the  FCR  (of  26  trees),  eight  for  the 
HUD  (of  20  trees),  and  four  plus  two  partial  ^  trees  for  the  INS  (of  six  trees).  Trees  were 
selected  to  be  representative  in  terms  of  length  and  complexity.  Six  of  the  FCR  fault  trees  were 
of  average  complexity,  and  two  were  above  average  in  complexity;  five  of  the  HUD  fault  trees 
were  average,  two  were  complex,  and  one  was  simple;  and  four  of  the  INS  fault  trees  were 
average  and  two  were  complex.  The  partial  fault  trees  were  of  average  complexity.  Each  tree 
was  analyzed  by  identifying  all  faults  covered  by  the  tree,  then  identifying  the  maintenance 
actions  (tests  and  rectifications)  performed  to  isolate  each  fault.  The  number  of  maintenance 
actions  and  parts  used  to  isolate  each  fault  was  recorded.  A  similar  process  was  followed  for  the 
IMIS.  An  analyst  entered  the  IMIS  with  the  appropriate  symptom  for  each  fault.  He  then 
entered  the  appropriate  information  until  he  reached  the  rectification  for  each  fault.  The  number 
of  maintenance  actions  and  rectifications  were  recorded.  The  results  of  this  analysis  are 
presented  in  Table  25. 

The  table  presents  the  number  of  unnecessary  rectifications  (part  replacement  or  wire 
repairs)  made  in  the  process  of  isolating  each  fault.  Examination  of  the  table  indicates  large 
differences  in  the  effectiveness  of  the  paper  TO  fault  trees  and  the  IMIS  diagnostics  (in  terms  of 
unnecessary  rectifications  made)  for  the  three  subsystems.  The  IMIS  appears  to  have  the  most 
benefit  for  the  INS  subsystem  where  unnecessary  replacements  were  reduced  by  87.5  percent 
and  for  the  FCR  with  a  reduction  of  51.6  percent.  The  IMIS  also  reduces  unnecessary 
rectifications  for  the  HUD,  however  only  by  9.5  percent. 

The  analysis  identifies  the  number  of  unnecessary  part  removals  for  locating  each  fauh 
covered  by  the  trees  one  time.  However,  faults  occur  with  different  frequencies.  Thus,  this 
information  does  not  provide  a  complete  picture  of  the  capability  of  the  IMIS  to  reduce  the 
number  of  unnecessary  rectifications.  A  second  analysis  was  undertaken  to  provide  an  estimate 
based  upon  the  expected  frequency  of  occurrence  of  each  fault.  A  comparison  was  made  by 
estimating  the  number  of  unnecessary  part  rectifications  which  would  occur  in  100  repetitions  of 
each  of  the  fault  trees  listed  in  Table  25  using  the  paper  TO  and  100  repetitions  using  fMIS.^"* 

The  unnecessary  replacement  estimates  were  based  upon  the  probabilities  of  the 
occurrence  of  each  fault  and  the  probabilities  of  each  test  passing  or  failing.  The  probabilities 
used  were  from  the  IMIS  data  base.  They  were  generated  by  engineers  at  Lockheed 
Corporation,  Fort  Worth  Division,  prime  contractor  for  the  F-16  and  authors  of  the  F-16  TOs. 
The  probabilities  were  based  on  estimates  and  computations  provided  by  engineers  who  designed 
the  test-bed  subsystems,  inputs  from  experienced  FCR,  HUD,  and  INS  technicians,  and 
component  failure  data  from  the  Air  Force  Maintenance  Data  Collection  System.  The 
rectification  estimates  were  made  by  following  the  fault  trees  and  applying  the  appropriate 
probabilities  at  each  point.  An  example  is  presented  in  Figure  5. 


The  partial  fault  trees  were  "off  shoots"  of  a  larger  more  complex  fault  tree  provided  to 
evaluate  the  results  of  the  inspection  of  the  contents  of  a  memory  location  on  the  inertial 
navigation  unit  (INU).  They  accounted  for  only  a  portion  of  the  possible  causes  of  the  main  tree. 
For  the  memory  inspect  trees,  this  is  100  repetitions  of  the  parent  tree. 
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Table  25.  Number  of  Unnecessary  Parts  Replaced  to  Isolate  Each  Fault 


One  Time  for  Representative  FCR,  HUD,  and  INS  Fault  Trees 


Unnecessary  Parts 

Fault  Tree 

Rectifications 

TO 

IMIS 

Difference 

FCR 

XD 

11 

4 

4 

0 

AL 

19 

18 

6 

12 

AD 

4 

0 

0 

0 

AE 

4 

0 

0 

0 

XS 

4 

3 

2 

1 

XMOl 

6 

2 

1 

1 

AN 

3 

2 

1 

1 

AU 

3 

2 

1 

1 

Total  FCR 

54 

31 

15 

16 

Percent  Change 

51.6 

HUD 

AL 

6 

9 

5 

4 

BHOl 

5 

3 

3 

0 

XD 

6 

0 

2 

-2 

AK 

3 

3 

3 

0 

AD 

4 

0 

0 

0 

AE 

4 

0 

0 

0 

BM 

5 

3 

3 

0 

BR 

5 

3 

3 

0 

Total  HUD 

38 

21 

19 

2 

Percent  Change 

9.5 

INS 

AF 

11 

19 

1 

18 

XD 

8 

12 

2 

10 

XG 

6 

9 

1 

8 

AGXD 

11 

29 

2 

27 

AG  144/200 

5 

20 

3 

17 

AG  145/100 

4 

15 

4 

11 

Total  INS 

45 

104 

13 

91 

Percent  Change 

87.5 

Figure  5  illustrates  how  the  rectification  estimates  were  made.  The  fault  tree  provided  in 
the  paper  TO  (FCR  fault  tree  AN)  starts  with  running  the  subsystem  BIT  to  verify  the  fault  and 
obtain  the  MFL.  If  there  is  a  fault.  Test  1  always  "fails"  (p  =  1 .0).  The  fault  tree  then  prescribes 
Rectification  1  (replace  antenna).  Thus,  the  antenna  is  always  replaced,  requiring  100  antennas 
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Paper  TO  Fault  Tree  AN  IMIS  Fault  Tree  AN 

for  Fire  Control  Radar  for  Fire  Control  Radar 

(Electronic  Equivilent 


Probability  MTTR 

Test1  1.00  14 

Test  2  .08  5 

Rectectification  1  .76  75 

Rectectification  2  .08  90 

Rectectification  3  .16  46 


MTTR  100  Repetitions 

TO  104.76  Minutes 
IMIS  88.56  Minutes 


MTTR  =  Mean  Time  to  Repair 
R  =  Rectification 
T  =  Test 


Rectifications  Required  for  100 
Repetitions  of  Fault  Tree  AN 


Paper  TO 
Red  100 
Rec2  8 
Rec3  16 

IMIS 

Rec  1  92 

Rec2  8 
Rec  3  16 


1.00x100 

.08x100 

.16x100 


(.76+.16)x100 

.08x100 

.16x100 


Parts  Required  for  100 
Repetitions: 

Paper  TO  124 
IMIS  116 


Figure  5.  Example  Paper  TO  and  IMIS  Fault  Trees  with  Calculations  of  Estimated 
Rectifications  for  100  Repetitions  of  Each  Fault  Tree 
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in  100  repetitions.  Replacing  the  antenna  rectifies  the  fault  76  percent  of  the  time.  If  replacing 
the  antenna  does  not  fix  the  problem.  Test  2  is  required.  The  answer  to  Test  2  is  yes  8  percent 
of  the  time,  leading  to  Rectification  2  (wiring  repair),  requiring  8  repairs  in  100  replications.  If 
the  Test  2  answer  is  no.  Rectification  3  (replace  PSP)  is  made,  requiring  16  more  parts.  Thus, 
100  repetitions  of  paper  TO  Fault  Tree  AN  require  124  rectifications:  100  to  fix  the  problem 
and  24  unnecessary  rectifications. 

The  IMIS  provides  a  different  strategy.  Test  1  is  omitted  (this  information  is  obtained 
from  the  pilot  during  the  debrief  using  IMIS  or  during  fault  verification  by  the  technician).  IMIS 
requires  performing  Test  2  first.  If  the  answer  to  Test  2  is  yes.  Rectification  2  (repair  wiring)  is 
required,  and  the  antenna  and  PSP  are  eliminated  from  consideration.  If  the  Test  2  answer  is  no, 
the  antenna  and  PSP  are  both  suspect.  Since,  there  is  no  test  which  will  eliminate  either  the 
antenna  or  PSP  from  consideration  (an  ambiguity  group),  the  only  option  remaining  is  to  replace 
the  antenna  or  PSP  and  verify  whether  the  fault  still  exists  in  the  subsystem.  The  IMIS 
recommends  replacing  the  antenna  because  it  has  the  highest  probability  of  failure.  If  the  fault  is 
still  present,  it  is  in  the  PSP.  IMIS  requires  eight  wiring  repairs,  92  antenna  replacements  (76 
times  when  it  is  bad  and  16  times  en  route  to  the  PSP),  and  16  PSPs  for  a  total  of  1 16 
rectifications.  Thus,  IMIS  requires  eight  fewer  rectifications  than  the  paper  TO  for  100 
repetitions. 

The  same  analysis  procedure  can  be  used  to  estimate  the  times  for  completing  the 
rectifications  using  the  paper  TO  and  IMIS.  The  Air  Force  has  established  mean  time  to  repair 
(MTTR)  estimates  for  every  task  required  for  the  test-bed  subsystems.  The  estimated  M  l  I  K  for 
100  repetitions  equals  the  sum  of  the  fi'equency  of  each  test  and  rectification  times  the  associated 
MTTR.  The  MTTRs  were  based  on  use  of  the  paper  TO  to  make  the  repairs.  These  same 
MTTRs  were  used  to  estimate  the  rectification  times  for  the  IMIS.  Thus,  the  estimates  do  not 
consider  any  differences  in  performance  times  for  each  repair  which  would  result  from  using  the 
IMIS.  The  actual  repair  times  are  not  expected  to  be  very  different,  whether  the  IMIS  or  paper 
TO  is  used.  Thus,  the  MTTRs  provide  a  reasonable  estimate  of  performance  times  with  IMIS. 
However,  the  MTTR  based  estimates  do  not  reflect  the  time  advantages  of  the  use  of  the  IMIS 
to  order  parts.  A  second  set  of  time  estimates  was  made  for  the  IMIS  condition  to  incorporate 
the  advantages  of  IMIS  part  ordering. 

The  estimated  unnecessary  rectifications  for  each  of  the  fault  trees  analyzed  are  presented 
in  Table  26.  The  table  provides  the  total  number  of  rectifications  for  the  fault  trees  studied 
broken  down  by  type.  Three  types  of  rectifications  are  shown:  major,  minor,  and  repair. 

Major  —  Replace  a  primary  LRU  such  as  the  PDF  or  INU. 

Minor  —  Replace  a  minor  (inexpensive)  component  such  as  a  relay. 

Repair  —  A  minor  on-equipment  repair  such  as  a  wiring  repair. 

The  data  provided  in  Table  26  provides  further  support  for  the  observation  that  the 
benefits  of  the  IMIS  diagnostics  vary  with  the  subsystem.  The  IMIS  yields  a  substantial  savings 
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Table  26.  Total  Number  of  Unnecessary  Rectification  by  Subsystem  for  100 
Repetitions  of  Each  Fault  Tree  and  the  Percent  Reduction  in  Parts  Requirements 

Unnecessary  Rectifications  Rectifications  Percent 

Subsystem _ TO _ IMIS _ Saved _ Improvement 

FCR 

Major  127.0  79.0  48.0  38.2 

Minor  45.5  36.0  9.5  20.9 

Repair  2.4  21,0  -18.6  -779.0 

HUD 

Major  97.0  94.0  3.0  3.09 

Minor  0.0  0.0  0.0  0.0 

Repair  0.0  0.0  0.0  0.0 

INS 

Major  289.0  74.0  215.0  74.4 

Minor  0.0  0.0  0.0  0.0 

Repair  433.0  47.0  386.0  89.1 

Total 

Major  513.0  247.0  266.0  51.9 

Minor  46.0  36.0  10.0  20.9 

Repair  435.0  68.1  366.9  84.4 

in  parts  for  the  INS  and  FCR  subsystems.  However,  the  IMIS  diagnostics  do  not  appear  to  have 
much  benefit  for  the  HUD.  Further  study  is  needed  to  determine  what  characteristics  influence 
the  ability  of  the  IMIS  to  reduce  the  number  of  unnecessary  removals 

The  MTTR  Estimates  are  presented  in  Table  27.  The  estimates  represent  an  estimate  of 
the  average  time  to  complete  each  of  the  fault  trees  studied  100  times.  This  data  supports  the 
observation  that  the  IMIS  diagnostics  have  the  greatest  benefit  for  the  INS. 

Table  27.  Mean  Times  to  Repair  (in  Minutes)  for 
Each  of  100  Repetitions  of  Each  Fault  Tree 
_ (Adjusted  for  Part  Ordering  Time) _ 

Subsystem _ TO _ IMIS 

FCR  108.1  100.7 

HUD  78.3  72.6 

INS  127.8  98.9 
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To  further  evaluate  the  benefits  of  the  IMIS  diagnostics,  the  analysis  was  extended  to 
estimate  the  potential  cost  savings.  Cost  savings  were  estimated  by  multiplying  the  number  of 
units  of  each  LRU  saved  by  the  cost  to  process  the  good  unit  through  the  depot  checkout  and 
repair  process.  The  cost  estimates  are  summarized  in  Table  28. 

The  cost  data  presented  in  Table  28  indicates  that  major  cost  savings  can  be  achieved 
with  the  implementation  of  the  BVUS.  Cost  savings  result  from  both  a  reduction  in  expenditures 


Table  28.  Estimated  Savings  in  Cost  of  Unnecessary  Rectifications  Resulting  from  the  Use  of 
_ IMIS  in  Place  of  Paper  TOs  (for  100  Repetitions  of  Each  Fault  Tree  Evaluated) 


Subsystem  Rectification 

Number 
of  Units 
Saved 

Unit 
Cost 
in  $ 

Unit 

Hours 

Saved 

Projected 
$  Cost 
Savings 

Hours 

Saved 

(Lost) 

FCR 

Replace  Antenna 

8 

18,380 

95 

147,040 

13 

Replace  DMT 

-1.1 

39,580 

74 

-43,538 

(1) 

Replace  MLPRF 

30.8 

47,075 

68 

1,449,910 

35 

Replace  Power  Relay  K2 

8 

500* 

60 

4,000 

8 

Replace  PSP 

1.4 

79,840 

66 

111,776 

2 

Replace  RDR  Power  Relay  K1 

10.5 

500 

60 

5,250 

11 

Repair  AC  Compression 

3.5 

0 

90 

0 

5 

Repair  Wiring 

-7.5 

0 

90 

0 

(11) 

Total  FCR  Savings  -  Parts 

57.6 

1,674,438 

66 

Total  FCR  Savings  -  Repairs 

(4) 

0 

(6) 

HUD 

Replace  EU 

11.5 

14,445 

45 

166,118 

9 

Replace  GAC 

-4.5 

16,960 

44 

(76,320) 

(3) 

Replace  PDU 

-4.8 

20,925 

58 

(100,440) 

(5) 

Total  HUD  Savings  (Loss)  -  Parts 

2 

(10,643) 

1 

INS 

Replace  ADI 

4 

5,000* 

72 

20,000 

5 

Replace  HSI 

1 

5,000* 

72 

5,000 

1 

Replace  INU  (3) 

183 

24,270 

74 

4,441,410 

226 

Replace  PSP 

4 

79,840 

76 

319,360 

5 

Replace  Avionics  Power  Panel 

54 

5,000* 

72 

270,000 

65 

Repair  MUX  Address  Jumpers 

82 

no 

0 

150 

Align  INU 

100 

15 

0 

25 

Total  INS  Savings  Parts 

246 

5,055,770 

302 

Total  INS  Repairs/Alignments 

182 

175 

Totals 

Part  Savings 

306 

6,719,566 

369 

Repair/ Alignment  Savings 

178 

0 

169 

*  Estimated  cost 
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for  components  and  a  reduction  in  man-hours  required  to  make  unnecessary  rectifications.  From 
the  data,  it  is  clear  that  the  cost  benefits  vary  greatly  from  subsystem  to  subsystem. 
Implementation  of  the  IMIS  for  the  INS  would  result  in  substantial  savings  (over  five  million 
dollars  for  100  occurrences  of  the  fault  codes  associated  with  the  six  fault  trees  studied).  The 
data  also  indicates  significant  savings  can  be  achieved  by  implementing  IMIS  diagnostics  forthe 
FCR.  However,  the  data  indicates  that  the  IMIS  diagnostics  would  not  significantly  reduce  costs 
for  the  HUD.  For  the  eight  fault  trees  studied,  the  use  of  IMIS  would  have  resulted  in  a  slight 
cost  penalty. 


Other  Findings 


Helps  Provided 

Although  helps  given  were  not  used  as  a  formal  measure  for  the  study,  the  number  of 
helps  required  by  the  technicians  provides  a  good  indicator  of  the  ease  of  use  of  the  IMIS  versus 
the  paper  TO  and  of  the  relative  ability  of  the  IMIS  and  paper  TO  to  meet  the  technicians’ 
information  needs. 

For  analysis,  helps  were  categorized  by  impact  and  by  type  of  help  given.  The  impact 
categories  were:  none,  minor,  and  major. 

None  —  help  was  informative  and  had  no  impact  on  outcome  of  session. 

Minor  —  help  was  given  as  a  positive  reinforcement  or  to  boost  confidence  where  the 
technician  was  demonstrating  apparent  confusion  or  fhistration. 

Major  —  help  was  given  in  response  to  a  direct  question  after  the  technician  had  reached 
a  dead  end,  or  was  confused  or  fmstrated  beyond  independent  recovery.  The 
help  had  a  definite  impact  on  the  session  outcome. 

Four  types  of  helps  were  given.  These  types  are:  system  knowledge  or  background, 
problem  solving,  clarify/supplement  technical  data,  and  technical  data  use. 

System  Knowledge  or  Background  —  provided  primarily  for  APG  technicians  to 

compensate  for  a  lack  of  experience  with  the  systems  (e.g.,  system  fianction  and 
interaction). 

Problem  Solving  —  provided  to  help  technicians  interpret  information  such  as  cockpit 
displays,  aircraft  response,  or  MFLs. 

Clarify/Supplement  Technical  Data  —  provided  where  the  technical  data  omitted 

necessary  information  such  as  configuration  data  or  marking  of  connector  pins. 
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Technical  Data  Use  —  provided  to  help  technicians  find  and  use  the  required  information 
in  the  paper  TO  or  IMIS. 

The  total  helps  provided  are  summarized  in  Table  29. 

Examination  of  Table  29  reveals  the  expected  pattern.  The  great  majority  of  helps  were 
provided  to  the  APG  technicians,  and  most  of  those  helps  were  required  when  the  technicians 
used  the  paper  TO.  The  APG  technicians  required  84  percent  of  the  total  helps,  and  65  percent 
of  the  helps  occurred  when  using  the  paper  TO.  Most  of  the  helps  given  to  APG  techmcians 
were  provided  during  their  first  few  problems.  The  specialists  required  veiy  few  helps;  slightly 
more  of  those  helps  occurred  when  using  the  TO. 

The  system  knowledge  helps  appear  to  have  had  little  impact  on  the  outcome.  All  of 
these  helps  were  judged  as  having  no  impact  or  only  a  minor  impact.  Most  of  the  system 
knowledge  helps  were  given  to  the  APG  technicians.  Many  of  these  helps  were  system 
orientation  information,  provided  to  satisfy  the  technicians’  curiosity  about  how  the  subsystems 
work.  They  indirectly  influenced  performance  because  they  gave  the  technicians  a  better 
knowledge  base  from  which  to  attack  the  problems  assigned. 

Table  29.  Total  Number  of  Helps  by  Category  for  Specialists  and  APG  Technicians  by 


Degree  of  Impact  on  Problem  Outcome 


Impact 

_ Wi. 

System 
Knowledge 
TO  IMIS 

_ * . 

Problem 

Solving 

TO  IMIS 

Clarify/ 
Supplement 
TO  IMIS 

Tech  Data 
Use 

TO  IMIS 

Total  Helps 
TO  IMIS 

Specialist 

No  Impact 

1 

6 

0 

0 

0 

0 

0 

0 

1 

6 

Minor 

0 

1 

0 

0 

2 

6 

15 

3 

17 

10 

Major 

0 

0 

0 

0 

0 

0 

7 

5 

7 

5 

Total 

1 

7 

0 

0 

2 

6 

22 

8 

25 

21 

APG 

Technician 

No  Impact 

24 

28 

1 

3 

0 

0 

0 

0 

25 

31 

Minor 

6 

0 

10 

9 

18 

22 

72 

13 

106 

44 

Major 

0 

0 

1 

0 

1 

0 

31 

5 

33 

5 

Total 

30 

28 

12 

12 

19 

22 

103 

18 

164 

80 

Totals 

No  Impact 

25 

34 

1 

3 

0 

0 

0 

0 

26 

37 

Minor 

6 

1 

10 

9 

20 

28 

87 

16 

123 

54 

Major 

0 

0 

1 

0 

2 

0 

38 

10 

40 

10 

Total 

31 

35 

12 

12 

21 

28 

121 

26 

189 

101 

52 


Problem  solving  helps  appeared  to  have  only  a  moderate  impact  on  the  test  outcome. 

Most  of  the  helps  were  in  the  minor  category  and  were  largely  coaching  in  nature  (e  g.,  "where 
would  be  a  good  place  to  find  that  kind  of  information"  or  "remember  what  you  learned  in 
training").  Most  of  the  helps  were  given  to  the  APG  technicians  who  were  unaccustomed  to 
performing  troubleshooting  tasks  and  were  not  comfortable  with  this  type  of  problem  solving. 
Specialists  had  little  problem  in  this  area  because  troubleshooting  tasks  are  a  major  component  of 
their  day-to-day  work  requirements. 

Use  of  the  technical  data  was  the  primary  problem  encountered  by  both  specialists  and 
APG  technicians.  Problems  encountered  in  using  the  TO  accounted  for  most  of  these  helps.  Of 
the  151  helps  for  technical  data  use,  125  (83  percent)  were  for  using  the  TO.  As  would  be 
expected,  most  of  the  technical  data  use  helps  were  for  the  APG  technicians,  and  85  percent  of 
these  were  for  help  in  using  the  TO.  Thirty-one  of  these  were  classified  as  major;  most  were  for 
navigating  through  the  paper  TOs  and  using  the  wiring  diagrams.  The  specialists  also 
encountered  some  difficulties  in  using  both  types  of  technical  data.  The  primary  problem 
encountered  in  using  the  TO  was  using  the  wiring  diagrams;  the  primary  problem  encountered 
with  the  IMIS  was  failure  to  read  and  follow  instructions. 

Impact  of  IMIS  Demonstration  System  Speed 

As  indicated  earlier  in  this  report,  one  limitation  of  the  IMIS  Demonstration  System  is  the 
time  for  the  system  to  process  and  present  technical  data.  System  response  time  ranges  from  one 
or  two  seconds  (e.g.,  to  present  the  next  step  in  a  sequence)  to  more  than  one  minute  (e.g.,  to 
recalculate  the  set  of  probable  faults).  The  slow  response  times  slowed  technician  performance 
noticeably  for  some  subtasks.  An  IMIS  system  for  operational  implementation  is  not  expected  to 
have  these  speed  limitations.  Consequently,  it  is  likely  that  the  time  estimates  developed  for 
IMIS  in  this  field  test  are  significantly  greater  than  will  be  experienced  with  an  operational 
system.  Thus,  performance  estimates  based  upon  the  IMIS  Demonstration  System  will 
underestimate  the  benefits  of  an  IMIS,  unless  the  data  is  adjusted  to  allow  for  improved  system 
performance.  To  provide  an  indication  of  how  much  technician  performance  was  slowed  by  the 
IMIS  Demonstration  System,  an  informal  comparison  was  made  with  a  "second  generation" 

EVAS  under  development  by  the  AL/HRGO  in  an  in-house  project. 

The  "second  generation"  IMIS  —  the  lETM  Presentation  System  (IPS)  —  is  a  package 
of  DOSAVindows-based  software  capable  of  performing  most  of  the  functions  performed  on  the 
IMIS  Demonstration  System.  IPS  is  designed  to  run  on  the  IMIS  Demonstration  System  PMA. 
In  addition,  it  will  operate  on  most  PCs  with  at  least  eight  megabytes  of  RAM  and  operating 
Microsoft  Windows  3.1.  The  IPS  uses  a  simpler  and  more  efficient  software  design.  The  times 
required  for  the  IPS  to  present  information  are  a  small  fraction  of  the  times  required  for  the  IMIS 
Demonstration  System  used  in  the  field  test.  The  EPS  is  capable  of  responding  to  most 
information  requests  in  less  than  one  second,  with  no  information  request  requiring  more  than 
15  seconds  (to  recalculate  the  set  of  probable  faults). 

Rough  estimates  of  the  potential  benefit  of  the  speed  improvements  provided  by  the  IPS 
were  developed  by  recording  the  times  required  for  "paging  through"  the  problems,  requesting 
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information  and  responding  to  prompts  in  the  same  way  that  technicians  did  when  actually 
performing  the  problems.  As  soon  as  the  computer  presented  a  screen  of  information,  the 
experimenter  immediately  pressed  next  or  other  appropriate  response  so  that  almost  all  time 
recorded  was  due  to  computer  processing  time.  It  was  assumed  that  experimenter  response 
times  would  be  the  same  for  both  computer  systems.  Times  were  recorded  for  eight  problems 
using  the  IMIS  Demonstration  System  and  six  problems  using  IPS.  It  was  not  possible  to 
directly  estimate  IPS  times  for  the  INS  problems  because  the  IPS  memory  inspect  capability  was 
not  available.  The  mean  times  for  the  FCR  and  HUD  problems  were  used  to  estimate  the  times 
for  those  subsystems,  respectively.  The  mean  times  for  the  FCR  and  HUD  problems  combined 
were  use  to  estimate  the  INS  problem  times.  A  conservative  adjustment  time  was  then  estimated 
and  used  to  adjust  the  data  collected  in  the  test  for  the  total  problem  time.  The  time  estimates 
developed  are  presented  in  Table  30. 


Table  30.  Comparative  Speeds  of  IMIS  Demonstration 


System  (IMIS  PS)  and  PS 


Test-Bed  Subsystem 

IMIS  DS 

IPS 

Percent 

Improvement 

Fire  Control  Radar 

13:26 

1:39 

8S 

Heads-Up  Display 

10:21 

1:39 

84 

Inertial  Navigation  Sys. 

16.34 

2.29 

85 

The  estimates  developed  suggest  that,  if  IPS  had  been  used  for  the  test,  performance 
times  would  have  been  reduced  by  from  eight  to  14  minutes.  The  IPS  processing  speeds  are 
believed  to  be  representative  of  the  speed  of  a  system  fielded  for  operational  use.*^ 

NASA  TLX  Workload  Ratings 

The  NASA  TLX  ratings  are  presented  in  Table  3 1 .  The  workload  ratings  made  by  the 
technicians  after  each  problem  were  significantly  higher  following  performance  of  a  task  with  the 
TO  than  ratings  given  following  performance  of  a  task  with  the  IMIS  (p  <  .001).  APG 
technicians  rated  workload  higher  than  specialists  (p  <  .001).  This  finding  is  consistent  with 
expectations  because  the  tasks  were  new  to  the  APG  technicians  and  represented  more  of  a 
challenge  to  them  than  to  the  specialists.  APG  technicians  tended  to  rate  the  paper  TO-based 
tasks  much  higher  than  did  the  specialists,  while  there  was  only  a  small  difference  in  the  ratings 


The  IPS  is  not  a  fully  developed  system.  Work  on  the  system  was  halted  when  it  was 
determined  that  the  IMIS  Demonstration  System  would  be  used  for  the  field  test.  However,  fiar 
the  IPS,  most  of  the  capabilities  required  to  support  the  field  test  are  operational  and  suitable  for 
use  as  a  comparison  system.  The  only  capabilities  required  for  the  field  test  that  are  not 
complete  are  the  1553  interface  and  the  RF  Link. 
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Table  31.  Mean  NTLX  Ratings 


TO 

IMIS 

Avionics  Specialists 

35.36 

23.56 

APG  Technician 

48.90 

28.41 

Total 

42.13 

25.98 

by  the  specialists  and  APG  technicians  following  tasks  using  the  IMIS.  In  addition,  there  was  an 
interaction  of  media  and  specialty  (p  <  .015).  For  paper  tasks,  the  APG  technicians  rated 
workload  significantly  higher  than  the  specialists  ( p  <  .001).  For  IMIS  tasks,  there  was  no 
significant  difference  in  workload  ratings  by  specialists  and  APG  technicians.  Thus,  the  IMIS 
had  a  greater  impact  on  the  workload  of  the  APG  technician,  as  reflected  in  a  significant 
interaction  (p  <  .001)  between  media  and  specialty. 

IMIS  Characteristics  Questionnaire 

After  completing  all  assigned  tasks  with  the  IMIS,  the  technicians  completed  a  69-item 
questionnaire.  The  questionnaire  was  designed  to  measure  technician  reactions  to  various 
characteristics  of  the  IMIS  Demonstration  System.  Technicians  responded  on  a  seven-point 
scale  to  questions  about  the  IMIS.  The  questions,  the  descriptors  which  the  technicians  based 
their  answers  upon,  and  the  mean  ratings  given  each  question  are  presented  in  Table  32.  The 
ratings  were  evaluated  on  a  seven-point  scale:  a  rating  of  one  generally  indicated  a  response 
very  favorable  to  IMIS,  a  rating  of  seven  generally  indicated  a  response  very  unfavorable  to 
IMIS.  The  questions  are  presented  in  rank  order,  with  the  most  positively  rated  questions  listed 
first.  In  interpreting  the  results,  note  that  the  technicians  were  responding  with  the  IMIS 
Demonstration  System  as  the  point  of  reference,  not  the  IMIS  concept  itself 

The  technicians  gave  an  average  rating  of  2.78  on  the  69  questions.  The  ratings  ranged 
from  1 .39  (indicating  part  ordering  is  easy  with  IMIS)  to  5.48  (indicating  that  the  response  time 
after  a  key  press  was  too  slow).  The  most  positively  rated  features  of  IMIS  were  its  automated 
parts  ordering,  work  order  close-out,  and  ability  to  use  the  IMIS  and  the  1553  interface  to 
operate  the  aircraft  BITs.  The  most  negatively  related  items  appear  to  be  related  to  the  slow 
speed  of  the  IMIS  Demonstration  System  and  a  perceived  need  for  a  full  keyboard.  The  system's 
slow  response  time  was  the  most  frequent  complaint  cited  on  the  exit  questionnaire;  this 
complaint  was  also  mentioned  during  informal  discussions  with  the  technicians  during  and  after 
their  participation  in  the  test. 

Exit  Questionnaire 

Review  of  the  written  comments  provided  by  the  technicians  at  the  end  of  the  test 
revealed  that  the  technicians  had  a  very  positive  attitude  toward  the  IMIS.  The  exit 
questionnaire  statements  clearly  indicate  that  the  technicians  like  the  concept  of  the  IMIS  and 
believe  it  has  great  potential.  This  observation  is  based  upon  statements  such  as: 
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Table  32.  IMIS  Characteristics  Questionnaire  Items  with  Mean  Ratings  for  Each  Item 


Question 

Descriptors 

Mean 

Rating 

The  process  of  ordering  a  part  from  the  IMIS 
system  was? 

Clear 

Unclear 

1.39 

Attaching  cables  to  the  PMA  was? 

Easy 

Difficult 

1.50 

Ordering  parts  through  IMIS  is  a? 

Good  idea 

Bad  idea 

1.64 

Closing  a  Work  Order  on  the  PMA  was? 

Easy 

Difficult 

1.70 

Time  permitted  to  become  familiar  with  the  PMA 
and  its  functions  was? 

Adequate 

Inadequate 

1.77 

The  BIT  process  and  procedure  was? 

Effective 

Ineffective 

1.88 

Self-tests  on  the  PMA  were? 

Effective 

Ineffective 

1.89 

The  size  of  the  PMA  screen  for  displaying  text  was? 

Adequate 

Not  adequate 

1.92 

Use  of  MFLs  in  the  Fault  Isolation  process  was? 

Helpful 

Not  helpful 

1.96 

The  training  you  received  was? 

Thorough 

Not  thorough 

2.00 

The  back  light  was? 

Helpful 

Not  helpful 

2.00 

Function  keys  (FI  through  F8)  were? 

Useful 

Not  useful 

2.04 

Filling  out  forms  while  using  IMIS  was? 

Easy 

Difficult 

2.09 

Connection  and  disconnection  to  the  1553  bus  was? 

Easy 

Difficult 

2.16 

Mapping  the  MFLs  to  the  fault  codes  was? 

Easy 

Difficult 

2.22 

The  ruggedness  of  the  keys  on  the  keypad  appeared 
to  be? 

Acceptable 

Not  acceptable 

2:23 

Highlighting  information  on  the  screen  made  tasks? 

Easier 

Harder 

2.25 
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Table  32.  (Continued) 


Question 

Descriptors 

Mean 

Rating 

In  comparison  with  paper  Tech  Orders,  the  IMIS 
fault  isolation  process  seemed? 

Better 

Worse 

125 

The  size  of  the  PMA  screen  for  displaying  graphical 
information  was? 

Adequate 

Not  adequate 

2.26 

Spacing  between  keys  on  the  keypad  was? 

Acceptable 

Unacceptable 

2.30 

In  troubleshooting,  the  diagnostic  aid  was? 

Helpful 

Not  helpful 

2.33 

Ability  to  override  the  IMIS-DM  recommended 
action  was? 

Easy 

Difficult 

2.38 

Failure  probabilities  associated  with  recommended 
actions  were? 

Helpful 

Not  helpful 

2.38 

Handling  of  CND's  on  the  PMA  was? 

Satisfactory 

Unsatisfactory 

2.43 

The  weight  of  the  PMA  was? 

Light 

Heavy 

2.43 

Instructions,  procedures,  and  guidance  about  1553 
connections  were? 

Effective 

Ineffective 

2.44 

In  troubleshooting,  the  diagnostic  block  diagrams 
were? 

Helpful 

Not  helpful 

2.45 

The  prop  or  handle  was? 

Effective 

Not  effective 

2.45 

Determining  how  to  proceed  through  the  PMA 
screens  in  order  to  support  your  task  was? 

Difficult 

Easy 

2.48 

The  width  and  length  (the  general  shape)  of  the 

PMA  was? 

Acceptable 

Not  acceptable 

2.52 

The  weight  of  the  PMA  was? 

Heavy 

Light 

5.57 

Message  lines  (informing  you  how  to  proceed  on  to 
the  next  piece  of  information)  were? 

Effective 

Ineffective 

2.59 
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Table  32,  (Continued) 


Mean 

Question _ Descriptors _ Rating 


The  PMA  screen  made  information? 

Easy  to  see 

Hard  to  see 

2.59 

Use  of  icons  was? 

Easy 

Hard 

2.60 

The  brightness  of  the  back  light  was? 

Good 

Bad 

2.61 

When  performing  maintenance,  the  size  of  the  text 
was? 

Easy  to  read 

Hard  to  read 

2.63 

On-line  help  on  the  PMA  would  be? 

Useful 

Not  useful 

2.65 

Scrolling  in  order  to  see  additional  procedural 
information  was? 

Good 

Bad 

2.68 

The  cursor  moved  where  you  thought  it  should? 

Always 

Never 

2.70 

Using  the  PMA  while  cables  were  attached,  affected 
PMA  use? 

Not  at  all 

Very 

negatively 

2.71 

The  IMIS  automatic  status  updating  would  reduce 
the  chatter  on  "bricks?" 

A  great  deal 

Not  at  all 

2.72 

Keeping  track  of  where  you  were  in  the  procedure 
was? 

Easy 

Difficult 

2.75 

When  viewing  Tech  Order  data,  location  and  labels 
on  function  keys  were? 

Appropriate 

Not 

appropriate 

2.77 

In  comparison  with  paper  Tech  Orders,  the  IMIS 
failure  data  seemed? 

More  accurate 

Less  accurate 

2.83 

Accessing  screens  required  to  perform  the  task  was? 

Easy 

Difficult 

2.83 

Zoom  capability  on  graphics  would  have  been? 

Helpful 

Not  helpful 

2.86 

Moving  the  cursor  around  the  screen  using  the  PMA 
arrow  keys  was? 

Easy 

Hard 

2.91 

Use  of  fault  codes  in  the  Fault  Isolation  process 
was? 

Helpful 

Not  helpful 

2.96 
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Table  32,  (Continued) 


Mean 

Question _ Descriptors _ Ratii^ 


Symbols  chosen  for  icon  pictures  were? 

Effective 

Not  Effective 

3,00 

Moving  the  pointer  around  the  screen  using  the 
PMA  thumb  knob  was? 

Easy 

Hard 

3.04 

The  recommended  "best  action"  seemed  to  be  an 
effective  choice  for  the  troubleshooting  task? 

Always 

Sometimes 

3.09 

The  keyboard  arrangement  used  on  the  PMA 
screen  was? 

Easy  to  use 

Hard  to  use 

3,17 

Reliability  of  the  IMIS  machine  that  you  worked 
with  was? 

Very  reliable 

Unreliable 

3.30 

Limited  descriptions  in  the  Discrepancy  field  of  the 
work  order  form  had? 

No  impact 

Limited  impact 

3.39 

Limited  descriptions  in  the  Discrepancy  field  of  the 
Work  Order  form  had? 

No  impact 

Negative 

impact 

3.39 

The  size  of  keys  on  the  keypad  were? 

Too  big 

Too  small 

3.55 

Handling  of  RTOK's  on  the  PMA  was? 

Satisfactory 

Unsatisfactory 

3.71 

Glare  on  the  screen  affected  performance  on  the 
task? 

Not  at  all 

A  great  deal 

3.79 

Limiting  the  presentation  of  steps  to  one  step  per 
screen  was? 

Necessary 

Not  necessary 

3.88 

The  limited  ability  to  describe  a  discrepancy 
impacted  the  FI  process? 

Not  at  all 

Negatively 

3.90 

Ability  to  see  and  read  the  screen  contents  from 
various  angles  was? 

Easy 

Hard 

3.96 

Pressing  and  activating  keys  on  the  keypad  was? 

Easy 

Hard 

4.00 

The  screen  was  readable  at? 

Many  angles 

Limited  angles 

4.23 
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Table  32.  (Concluded) 


Question 

Descriptors 

Mean 

Rating 

If  you  used  the  thumb  knob  and  the  arrow  keys, 
which  device  did  you  prefer? 

Arrow  keys 

Thumb  knob 

4.35 

A  full  keyboard  (allowing  use  of  all  alphabetical 
keys)  would  have  made  the  PMA  use? 

More  difficult 

Easier 

4.91 

Response  time  after  pressing  keys  was? 

Fast  enough 

Too  slow 

5.48 

Great  project  —  quite  a  bit  better  than  having  to  use  TOs.  I  hope  I  see  it  in  use 

somewhere,  sometime  before  I  get  out  of  the  service  or  even  in  the  civilian  world. 

IMIS  definitely  needs  to  be  an  integral  part  of  the  AF  [Air  Force].  The  AF  will  benefit 

greatly  from  the  implementation  of  IMIS. 

Very  good  unit.  If  it  were  not  for  the  waiting  on  the  computer,  it  would  be  great. 

There  are  several  features  of  the  IMIS  that  the  technicians  especially  like.  The  most 
frequently  mentioned  features  are  the  automated  parts  ordering,  integration  of  technical  data  (no 
need  to  carry  an  armload  of  TOs),  1553  interface,  simplification  of  the  troubleshooting  process, 
and  automated  work  order  close-out  procedure.  However,  there  are  also  several  features  of  the 
EVUS  Demonstration  System  they  did  not  like.  The  most  frequently  mentioned  problem  withihe 
IMIS  demonstration  speed  was  response  time.  The  majority  of  the  technicians  commented  about 
the  slow  system  response,  either  as  a  system  deficiency  or  a  needed  refinement.  The  IMIS 
Demonstration  System  user  interface  was  the  subject  of  several  comments.  The  primary  irritant 
from  the  user  interface  is  the  inconsistent  use  of  the  function  keys  and  select  keys  (e.g.,  "FI  OK" 
to  continue  versus  "F8  NEXT"  to  continue).  In  addition,  a  number  of  technicians  remarked 
about  the  Demonstration  System's  tendency  to  freeze  up,  indicating  that  system  reliability  must 
be  improved. 

The  technicians  provided  several  suggestions  to  improve  the  system.  These  included 
changes  such  as  providing  a  browsing  capability,  presenting  more  than  one  step  at  a  time,  and 
better  integration  of  the  diagnostic/rectification  processes  and  the  work  order  close-out  process 
so  that  most  of  the  information  required  for  completing  the  close-out  documentation  is 
automatically  generated. 

A  complete  listing  of  comments  made  on  the  exit  questionnaire  is  provided  in  Appendix  D. 
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DISCUSSION 


The  IMIS  field  test  was  designed  to  evaluate  the  capability  of  the  IMIS  to  support  the 
maintenance  diagnostic  and  repair  processes.  Earlier  studies  and  analyses  have  indicated  that  the 
IMIS  will  reduce  the  time  required  to  diagnose  and  repair  a  fault  in  an  aircraft  subsystem,  reduce 
the  consumption  of  spare  parts,  and  improve  the  accuracy  of  maintenance  data  reporting.  In 
addition,  the  studies  have  suggested  that  the  use  of  an  IMIS  will  make  it  possible  for  personnel 
with  less  extensive  training  on  a  subsystem  to  effectively  troubleshoot  and  repair  it.  This  field 
test  was  designed  to  examine  the  extent  to  which  the  IMIS  Demonstration  System  is  able  to 
achieve  these  expectations  and  the  desirability  of  implementing  an  IMIS  for  operational 
use.Impact  of  IMIS  on  Technician  Performance 

Impact  of  IMIS  on  Technician  Performance 

The  study  focused  upon  the  ability  of  the  IMIS  to  support  two  types  of  maintenance 
technicians:  avionics  specialists  and  non-avionics  specialists  (APG  technicians). 

•  Avionics  specialists  —  trained  in  the  general  principles  of  avionics  maintenance; 
possess  specific  training  and  experience  in  maintaining  the  F-16  avionics 
subsystems  used  as  test-beds  for  the  study. 

•  Non-avionics  specialists  (APG  technicians)  —  possess  training  and  experience  in 
general  F-16  maintenance  procedures  but  have  no  specific  training  on,  or 
experience  with,  the  F-16  avionics  subsystems  used  as  test-beds  for  the  study. 

The  two  types  of  technicians  represented  both  ends  of  the  spectrum  of  skills  required  to 
maintain  the  test-bed  subsystems:  the  well-trained  avionics  specialist  with  experience  in 
maintaining  the  subsystems  and  the  APG  technician  with  general  maintenance  skills  but  no 
specific  training  or  experience  on  the  test-bed  subsystems.  This  section  of  the  report  examines 
the  test  results  with  regard  to  the  ability  of  the  IMIS  to  support  each  type  of  technician. 

IMIS  Support  for  the  Specialist 

The  avionics  specialist  technicians  were  expected  to  successfully  complete  a  high 
percentage  of  the  fault  isolation  problems,  using  either  the  TO  or  the  IMIS.  Their  performance 
met  this  expectation.  The  specialists  were  able  to  successfully  isolate  all  the  faults  using  either 
the  TO  or  IMIS.  In  addition,  they  were  able  to  successfully  complete*^  all  problems  using  the 
IMIS.  However,  they  were  able  to  successfully  complete  only  87  percent  of  the  problems  when 
using  the  TO.  The  failures  were  all  due  to  not  completing  all  checks  and  tests  required  to 
establish  system  health.  This  problem  appears  to  be  caused  by  the  way  system  health 
requirements  are  presented  in  the  TO,  which  makes  it  easy  to  overlook  key  requirements.  With 
the  IMIS,  it  is  not  possible  to  overlook  required  procedures.  They  are  automatically  presented  at 


Successful  completion  of  a  problem  was  defined  as  completing  all  required  subtasks, 
including  fault  verification,  fault  isolation,  rectification,  system  health,  and  documentation. 
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the  appropriate  point  in  the  process.  The  technician  must  perform  the  procedure  or  consciously 
choose  not  to  perform  it.  This  is  an  important  advantage  of  the  IMIS  for  both  efficiency  and 
safety  reasons. 

It  was  anticipated  that  specialist  technicians  would  be  able  to  perform  troubleshooting 
tasks  more  rapidly  when  using  the  IMIS  as  their  source  of  technical  data  than  when  using  paper 
TOs.  IMIS  was  expected  to  reduce  troubleshooting  times  by  providing  more  efficient  diagnostic 
routines  and  by  reducing  the  number  of  component  replacements  used  in  the  diagnostic  process. 
In  addition,  it  was  anticipated  that  additional  time  would  be  saved  by  the  IMIS  automatically 
inputting  the  close-out  documentation  to  CAMS. 

Time  is  saved  in  the  fault  isolation  process  by  minimizing  the  number  of  tests  and  by 
reducing  component  replacements  which  must  be  made.^^  Time  savings  resulting  from  reducing 
the  number  of  parts  unnecessarily  replaced  can  be  significant  because  each  replacement  requires 
ordering  the  part  from  supply  or  removing  a  good  part  from  a  nearby  aircraft.  Technicians  and 
supervisors  estimate  it  takes  the  technician  at  least  15  minutes  to  order  a  part.^^  The  IMIS 
automatically  provides  all  the  information  required  to  order  a  part  and  orders  the  part  via  the  RF 
link.  Thus,  the  15  minutes  (or  more)  required  to  order  a  part  is  saved  by  IMIS.  The  technician 
can  use  the  time  saved  to  remove  the  part  or  for  other  maintenance  activities  while  waiting  for 
the  part  to  be  delivered. 

The  data  collected  in  the  field  test  provides  support  for  the  assertion  that  the  IMIS  can 
significantly  reduce  the  time  required  to  diagnose  and  repair  faults  in  avionics  subsystems.  The 
use  of  the  IMIS  reduced  the  mean  times  to  perform  the  test  problems  by  approximately  27 
percent.  On  average,  the  technicians  required  approximately  26  minutes  longer  to  complete  the 
problems  using  the  TO.  However,  close  examination  of  the  data  indicates  that  the  primary 
source  of  the  savings  in  time  to  perform  the  task  was  due  to  the  use  of  the  IMIS  parts  ordering 
and  work  order  close-out  capabilities. 

The  data  demonstrates  that  the  IMIS  effectively  supports  avionics  specialists  performing 
diagnostic  tasks  on  the  test-bed  subsystems.  In  addition,  the  use  of  the  IMIS  can  result  in 
significant  time  savings,  reduce  errors,  and  reduce  parts  consumption. 

IMIS  Support  for  the  APG  Technician 

As  indicated  above,  APG  technicians  were  included  in  the  study  to  test  the  assertion  that, 
with  the  use  of  the  IMIS,  technicians  with  limited  subsystem  knowledge  can  successfully  perform 
fault  isolation  tasks  on  the  test-bed  subsystems.  APG  technicians  were  selected  for  the  non- 


The  diagnostic  procedures  provided  in  the  TO  frequently  require  replacement  of  a 
component  as  a  check  to  see  if  it  will  fix  the  problem.  As  a  result,  several  good  components 
may  be  replaced  before  the  bad  component  is  found. 

At  least  as  much  time  is  required  if  the  component  is  "borrowed"  from  a  nearby  aircraft 
because  the  technician  must  open  the  appropriate  aircraft  panel,  remove  the  component, 
annotate  the  aircraft  forms,  replace  the  part,  close  the  panel(s),  and  annotate  the  forms. 


specialist  role  because  they  receive  no  specific  training  on  the  test-bed  subsystems,  and  no  more 
than  familiarization  training  with  avionics  subsystems  in  general.  Thus,  it  was  believed  that  they 
would  represent  a  good  test  of  the  limits  of  the  IMIS  for  supporting  maintenance  by  personnel 
with  limited  training  on  the  test-bed  subsystems.  In  addition,  the  APG  technicians  were  familiar 
with  the  aircraft,  comfortable  working  around  it,  and  cockpit  qualified  (a  requirement  to 
participate  in  the  test). 

The  field  test  team  had  some  reservations  about  including  the  APG  technicians  in  the 
study.  There  was  confidence  that  the  technicians  would  be  able  to  do  at  least  a  creditable  job  on 
the  troubleshooting  problems  when  using  the  DVDS.  However,  it  was  uncertain  how  they  would 
do  using  the  paper  TOs.  These  concerns  were  alleviated  somewhat  during  the  pretest  when  an 
APG  technician  was  able  to  successfully  complete  all  the  required  tasks.  However,  it  became 
clear  after  the  first  two  test  subjects  APG  technicians  completed  their  problems  using  the  TO 
that  they  were  up  to  the  challenge.  The  technicians  started  their  first  TO  problems  with  obvious 
concern,  but  by  the  time  they  completed  the  first  two  problems,  their  anxiety  had  eased  and  their 
confidence  began  to  build,  as  did  their  subsystem  knowledge. 

The  APG  technicians  were  able  to  successfully  complete  99  percent  of  the  problems  with 
the  IMIS  and  69  percent  of  the  problems  with  the  paper  TO.  Thus,  when  they  used  the  IMIS, 
their  performance  on  this  measure  was  essentially  the  same  as  the  specialists  using  the  IMIS  and 
better  than  specialists  using  the  paper  TO.  Also,  when  using  the  IMIS,  the  APG  technicians’ 
overall  times  to  complete  the  problems  were  only  slightly  longer  than  the  specialists  times  using 
the  IMIS  and,  for  most  time  measures,  significantly  faster  than  the  times  for  the  specialists  when 
using  the  paper  TO.  The  relative  speed  of  performance  of  APG  technicians  using  IMIS  versus 
specialists  using  the  paper  TO  are  illustrated  in  Figure  6. 

The  APG  technicians  required  many  more  "helps"  to  get  through  the  problems.  The 
avionics  technicians  received  very  few  helps  (the  restrictions  on  providing  helps  were  much  more 
stringent  for  the  avionics  technicians).  More  helps  were  required  for  the  APG  technicians 
because  they  were  totally  unfamiliar  with  the  test-bed  subsystems  and  had  had  only  limited 
training  on  the  TOs  for  those  subsystems.  Helps  were  primarily  for  locating  subsystem 
components,  locating  and  interpreting  information  in  the  TOs,  and  using  or  reading  the 
multimeter.  Typically,  several  helps  were  required  for  an  APG  technician's  first  two  or  three 
problems.  Few,  if  any,  helps  were  required  after  they  had  completed  the  third  problem  and  were 
familiar  with  location  of  the  subsystem  components.  Nevertheless,  it  is  significant  that  the  APG 
technicians  were  able  to  successfully  isolate  faults  in  complex  avionics  subsystems  that  they  had 
never  worked  on  before.  A  high  degree  of  success  using  the  IMIS  was  expected.  The  level  of 
success  with  the  TO  was  something  of  a  surprise  because  there  was  some  concern  about  whether 
the  APG  technicians  would  be  able  to  solve  the  problems  at  all. 

One  objective  of  this  study  was  to  demonstrate  that,  using  the  IMIS,  APG  technicians 
could  troubleshoot  complex  systems  as  effectively  as  avionics  specialists  could  troubleshoot  the 
problems  when  using  the  TO.  This  objective  was  exceeded.  The  APG  technicians  were 
significantly  more  successful  in  performing  the  problems  with  the  IMIS  than  the  avionics 
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specialists  were  using  the  TO.  More  surprisingly,  there  was  no  significant  difference  in  the 
performance  of  the  APG  and  avionics  technicians  when  both  used  the  IMIS.  When  using  the 
IMIS,  the  APG  technicians  are  as  capable  as  the  avionics  specialists  in  performing  fault  isolation 
tasks  on  the  test-bed  subsystems.  However,  this  equality  applies  only  when  the  IMIS  provides 
adequate  fault  isolation  procedures.  It  is  unlikely  that  the  APG  technician  will  be  able  to  handle 
those  occasional  problems  that  are  not  resolved  using  the  diagnostic  procedures  provided  in  the 
technical  data.  The  special  training  and  experience  of  the  specialist  will  be  required  to  resolve 
these  problems. 

The  achievements  of  the  APG  technicians  when  using  the  IMIS  are  especially  noteworthy 
when  their  relative  inexperience  is  considered.  Most  of  the  APG  technicians  who  participated  in 
the  study  were  relatively  inexperienced.  Ten  of  the  APG  technicians  were  Skill  Level  3  with  less 
than  one  year  of  experience.  By  comparison,  the  specialists  were  relatively  experienced.  Eight 
were  Skill  Level  5  with  between  four  and  ten  years  of  experience,  and  four  were  Skill  Level  3 
with  between  12  and  18  months  of  experience.  Thus,  as  a  group,  the  APG  technicians  were 
relatively  inexperienced,  while  the  specialists  as  a  group  were  relatively  experienced. 
Nevertheless,  the  use  of  the  IMIS  made  it  possible  for  the  APG  technicians  to  perform  the 
troubleshooting  tasks  as  effectively  as  the  specialists. 

Parts  Savings  from  IMIS 

One  major  benefit  anticipated  from  the  IMIS  is  a  reduction  in  the  number  of  spare  parts 
used.  This  reduction  would  result  from  technicians  unnecessarily  replacing  fewer  "good"  parts 
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(due  to  either  the  diagnostic  strategy  used  or  technician  error).  Both  the  field  test  data  and  the 
fault  tree  analysis  provide  evidence  to  support  this  assertion.  The  mean  number  of  good  parts 
replaced  by  technicians  using  the  IMIS  was  significantly  less  than  the  mean  number  of  parts  used 
with  the  TO,  and  the  fault  tree  analysis  provided  clear  evidence  that,  for  at  least  two  of  the  test¬ 
bed  subsystems,  major  reductions  in  unnecessary  part  replacements  would  result  from 
implementation  of  the  IMIS.  The  data  also  reveal  some  of  the  limitations  of  the  IMIS 
diagnostics. 

The  data  clearly  illustrates  the  primary  limitations  of  the  IMIS  diagnostics  —  the 
limitations  imposed  by  the  technical  data  input  to  the  IMIS  and  the  design  of  the  subsystems 
under  test.  The  data  show  that  even  though  the  IMIS  diagnostic  algorithm  provides  the  most 
effective  sequencing  of  tests  to  isolate  a  given  fault,  unnecessary  component  replacements  will 
still  be  encountered  unless  the  subsystem  under  test  has  adequate  test  points  to  permit  isolating 
each  component  and  the  test  points  are  accurately  presented  in  the  technical  data.  The  F-16 
FCR,  HUD,  INS,  and  associated  fault  isolation  manuals  provide  good  examples  of  this  problem. 
Furthermore,  because  the  data  used  for  the  IMIS  diagnostics  was  based  on  the  fault  isolation 
manuals,  the  study  provides  a  good  example  of  how  IMIS  handles  the  problem,  given  the 
limitations  imposed  by  the  test-bed  subsystem  designs  and  the  test  procedures  provided  in  the 
paper  fault  isolation  manuals. 

The  diagnostic  strategy  used  in  F-16  fault  isolation  manuals  tends  to  replace  the 
component(s),  most  likely  to  cause  the  observed  symptom  first  with  little  attempt  to  verify  it  is 
bad  before  removing  it.  Then,  when  that  does  not  solve  the  problem,  the  technician  is  advised  to 
either  replace  more  components  or  conduct  tests  to  narrow  the  problem.  The  IMIS  takes  a  more 
systematic  approach  and  evaluates  the  alternatives  and  associated  costs  against  the  information 
received  from  each  action.  As  a  result,  if  there  is  an  inexpensive  test  that  could  eliminate  a 
component  from  consideration,  the  test  will  be  performed  first.  Problems  9  and  10  represent 
good  examples  of  the  two  approaches.  Both  replaced  the  most  probable  component  first 
because  there  was  no  way  to  eliminate  it  from  consideration.  When  this  did  not  fix  the  problem, 
the  TO  continued  replacing  components.  However,  the  IMIS  algorithm  recognized  an 
inexpensive  test  which  could  eliminate  the  two  remaining  suspect  components  from 
consideration.  IMIS  recommended  that  test  be  made  first.  The  test  led  to  the  cause  of  the  fault 
(wiring).  The  TO  recommended  replacing  three  good  components  before  recommending  the  test 
that  identified  the  problem.  The  result  was  three  good  parts  used  Avith  TOs  and  none  with  the 
IMIS. 


Most  good  parts  replaced  were  replaced  as  part  of  the  diagnostic  strategy;  the  remainder 
were  due  to  technician  error.  The  technical-data-directed  replacement  of  good  parts  appears  to 
be  the  result  of  two  factors:  (1)  the  diagnostic  philosophy  of  the  technical  data  developers  and 
(2)  the  design  of  the  avionics  subsystems  themselves.  The  aircraft  design  limits  the  development 
of  sound  diagnostic  procedures  to  some  extent.  If  the  design  does  not  provide  adequate  BIIs 
and  test  points,  the  developer  of  the  fault  isolation  manual  may  not  be  able  to  provide  adequate 
diagnostic  procedures.  However,  the  technical  data  writer  must  take  full  advantage  of  the 
available  (or  possible)  test  points  to  reduce  the  number  of  ambiguity  groups.  The  F-16  is  known 
for  having  many  ambiguity  groups.  When  ambiguity  groups  are  present,  the  fault  isolation 
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manual  provides  no  way  to  identify  the  faulty  component,  except  to  systematically  replace  the 
components  in  the  group  until  the  problem  is  fixed.  When  this  happens,  good  parts  are  replaced 
unnecessarily.  This  appears  to  be  the  primary  cause  of  the  unnecessary  use  of  good  parts  in  this 
study.  Unfortunately,  since  the  IMIS  technical  data  was  developed  from  the  fault  isolation 
manuals,  the  ambiguity  groups  existing  in  the  fault  isolation  manuals  were  also  reflected  in  the 
IMIS  data.  In  addition,  the  IMIS  developers  were  limited  to  the  tests  and  procedures  provided 
in  the  TOs.  As  a  result,  some  of  the  inadequacies  of  the  paper  fault  isolation  manuals  were 
carried  over  into  the  IMIS  data  base,  preventing  full  realization  of  the  benefits  of  the  IMIS 
diagnostics. 

It  is  unfortunate  that  program  constraints  prohibited  refinement  and  extension  of  the 
IMIS  technical  data  to  take  fUll  advantage  of  the  IMIS  diagnostics.  With  a  little  ingenuity  and 
effort,  a  good  technical  writer/analyst  often  can  identify  additional  tests  and  ways  of  breaking  up 
ambiguity  groups.  If  the  IMIS  analysts  had  been  permitted  to  break  up  the  ambiguity  groups  and 
modify  the  IMIS  technical  data  to  better  match  the  requirements  of  the  IMIS  diagnostic  module, 
it  is  likely  that  additional  savings  in  parts  would  have  been  achieved. 

The  fault  tree  analysis  revealed  another  important  observation  —  the  potential  of  the 
IMIS  diagnostics  for  reducing  unnecessary  part  replacements  is  dependent  upon  the  subsystem 
supported.  The  analysis  demonstrated  a  very  large  difference  in  the  ability  of  the  IMIS  to  reduce 
parts  consumption  for  the  INS  and  for  the  FCR.  The  analysis  indicated  a  potential  reductionin 
the  unnecessary  replacement  of  good  LRUs  of  approximately  74  percent  for  the  INS  and 
38  percent  for  the  FCR.  However,  the  estimated  savings  for  the  HUD  was  only  for  a  three 
percent  reduction.  The  reason  for  this  difference  could  not  be  determined  from  the  available 
information.  There  are  several  interrelated  factors  which  appear  to  contribute  to  the  difference  in 
impact.  These  include  the  complexity  of  the  subsystem,  the  number  of  LRUs  and  rectifications 
per  subsystem,  the  length  of  the  fault  trees,  and  the  characteristics  of  the  functions  performed  by 
the  subsystem  (e.g.,  information  exchange  versus  computation  intensive  functions).  Further 
study  is  needed  to  better  understand  this  issue. 

Support  for  Work  Order  Close-Out 

The  field  test  provided  limited  data  to  support  the  assertion  that  the  IMIS  will  reduce  the 
time  to  close-out  a  work  order  and  enter  the  data  into  the  CAMS.  An  adequate  comparison  of 
the  IMIS  and  the  current  procedures  for  work  order  close-out  and  data  input  was  not  possible 
due  to  two  problems: 


The  IMIS  technical  data  was  translated  directly  from  the  digital  version  of  the  fault 
isolation  manuals  to  the  format  required  by  the  IMIS.  Technical  content  was  not  changed. 
Changes  in  technical  content  were  not  permitted  because  the  intent  was  to  use  the  data  for 
operational  maintenance  activities  during  an  unconstrained  test,  originally  scheduled  to 
follow  the  IMIS  field  test.  Changes  in  the  technical  data  would  have  required  that  the  data 
be  reverified. 
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a.  It  was  not  possible  to  develop  a  good  measure  of  the  time  required  to  close-out 
the  work  order  and  input  the  information  to  CAMS,  since  the  live  CAMS  data 
base  could  not  be  used,  and  there  were  no  CAMS  terminals  available  with  access 
to  the  CAMS  training  data  base.  As  a  substitute,  the  technicians  recorded  the 
necessary  information  on  paper.  Time  to  record  the  information  was  recorded 
and  combined  with  an  estimate  of  the  time  required  to  gain  access  to  a  CAMS 
terminal  and  input  the  data. 

b.  The  IMIS  Demonstration  System  fully  implemented  the  IMIS  concept  for  the 
work  order  close-out  and  CAMS  input  processes.  The  IMIS  concept  (for  an 
operational  system)  provides  for  the  IMIS  to  automatically  record  most 
information  required  to  close-out  a  work  order.  All  the  technician  needs  to  do  is 
verify  that  the  information  on  the  close-out  form  is  correct  and  press  transmit. 
The  Demonstration  System  implementation  did  not  automatically  record  this 
information.  Thus,  the  technician  was  required  to  input  the  data  either  using  the 
keyboard  or  by  selecting  from  lists  of  alternatives. 

The  field  test  provided  some  limited  data  on  the  benefits  of  the  IMIS  for  supporting  the 
close-out  process.  The  data  indicates  that,  with  the  current  IMIS  Demonstration  System,  the 
time  to  close  a  work  order  and  input  the  data  to  CAMS  can  be  cut  in  half  When  the  close-out 
procedure  is  fully  automated  in  IMIS,  the  time  required  for  close  out  will  be  reduced  to  near 
zero,  a  savings  of  between  15  and  20  minutes. 

IMIS  Design  Considerations 

Although  the  IMIS  Demonstration  System  effectively  demonstrated  the  overall  IMIS 
concept,  there  are  several  deficiencies  in  the  system  which  must  be  avoided  in  any  system 
developed  for  operational  use.  These  deficiencies  are  discussed  briefly  below. 

Speed 


The  most  serious  deficiency  of  the  IMIS  Demonstration  System  is  the  slow  speed  with 
which  it  retrieves  and  presents  technical  and  management  information.  The  system  requires  from 
one  or  two  seconds  to  as  long  as  two  minutes  to  respond  to  an  information  request.  The  only 
process  which  is  reasonably  fast  is  moving  from  frame  to  frame  within  a  procedure,  where  all  the 
system  has  to  do  is  present  the  next  step  (without  graphics).  A  somewhat  more  complex  action, 
such  as  responding  to  an  input  from  a  branching  question,  requires  several  seconds,  typically 
three  to  eight.  Very  complex  actions,  such  as  updating  diagnostics  or  retrieving  data  from  a 
different  part  of  the  data  base,  may  take  one  minute  or  more. 

Although  these  times  do  not  seem  excessive  to  the  casual  observer,  they  do  become 
burdensome  as  one  gains  experience  in  using  the  system,  especially  when  working  under  time 
constraints.  The  technicians  who  participated  in  the  field  test  were  much  less  tolerant  of  the 
system's  slow  speed  than  were  the  managers  participating  in  the  End-to-End  Demonstration 
(EED).  The  technicians  in  the  field  test  used  the  system  for  several  hours,  under  time  pressure. 
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The  EED  participants  used  it  for  only  two  or  three  hours  and  not  under  pressure.  Improving  the 
speed  of  the  system  was  the  most  frequent  recommendation  made  by  the  field  test  participants 
for  improving  the  system.  An  IMIS  developed  for  operational  use  should  present  requested 
information  within  two  seconds  for  at  least  90  percent  of  the  requests.  No  request  should 
require  more  than  30  seconds  to  fill,  and  such  instances  should  be  rare. 

Human  Factors 

The  IMIS  Demonstration  System  user  interface  is  generally  well  done  and  provides  a 
good  starting  point  for  development  of  an  operational  system.  However,  there  is  considerable 
room  for  improvement.  The  field  test  has  highlighted  several  problem  areas  and  produced  some 
suggestions  from  the  technicians  for  improving  the  interface. 

One  major  problem  with  the  interface  is  its  lack  of  consistency  in  the  Demonstration 
System's  user  interface.  Next  to  system  speed,  the  most  frequent  complaint  from  the  technicians 
was  the  inconsistent  use  of  "FI,"  "F8,"  and  "Select"  to  make  a  selection.  In  some  places,  FI  is 
used  to  go  to  the  next  frame,  in  other  places,  F8  is  used  to  go  to  the  next  frame.  In  some  places 
FI  is  used  to  choose  from  a  menu,  in  other  places  SELECT  is  used  to  choose  from  a  menu. 

Other  inconsistencies  are  found  in  features  such  as  the  use  of  the  hour  glass  when  the  computer 
is  processing  and  the  use  of  prompts. 

The  technicians  recommended  improvements  such  as  presenting  more  than  one  step  at  a 
time  so  they  can  look  ahead  and  see  the  next  step,  providing  a  browsing  capability,  and  providing 
better  locator  information. 

The  IMIS  Demonstration  System  provides  a  good  starting  point  for  the  design  of  an 
IMIS  interface.  With  the  addition  of  a  few  additional  features,  correction  of  its  weak  points,  and 
refinement  of  its  rough  spots,  it  could  provide  an  effective  user  interface  —  assuming,  of  course, 
that  the  system's  slow  response  time  is  corrected. 

Designers  of  an  IMIS  for  operational  implementation  would  be  wise  to  carefully  study 
the  IMIS  Demonstration  System  user  interface.  It  provides  many  examples  of  good  practices  but 
also  offers  some  examples  of  awkward  and  ineffective  practices.  The  human  factors  engineers 
will  learn  a  lot  and  avoid  costly  errors  in  their  own  designs  by  spending  several  days  using  the 
system  to  perform  tasks,  by  observing  technicians  and  managers  using  the  system  for  various 
tasks,  and  by  talking  extensively  with  technicians  and  managers  who  have  used  the  system. 

Similar  studies  should  be  made  of  other  IMIS  like  systems  under  development.  See  Quill, 
Kander,  Wampler,  and  Wynkoop  (1995)  for  a  detailed  discussion  of  IMIS  human  factors  lessons 
learned. 
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Other  Considerations 


Technical  Data 

Proponents  of  the  IMIS  have  emphasized  its  capability  to  reduce  parts  consumption  and 
retest  OK  rates.  The  field  test  provided  support  for  this  benefit;  however,  the  benefits  were  not 
as  dramatic  as  expected.  There  is  reason  to  believe  that  the  inability  of  the  IMIS  to  more 
dramatically  demonstrate  this  benefit  is,  at  least  in  part,  due  to  limitations  of  the  technical  data 
used.  If  the  IMIS  analysts  had  had  the  freedom  to  develop  new  test  points,  which  the  IMIS 
algorithm  could  have,  in  turn,  taken  advantage  of,  it  is  likely  that  the  number  of  good  parts 
replaced  when  using  the  IMIS  would  have  been  reduced,  perhaps  dramatically.  If  additional  test 
points  are  not  possible,  there  is  no  better  way  than  to  replace  parts  until  the  problem  is  solved. 

In  that  case,  no  diagnostic-aiding  system  will  achieve  a  dramatic  reduction  in  the  use  of  parts. 

A  related  issue  is  the  quality  of  the  data  in  general.  If  the  data  input  to  IMIS  is  of  poor 
quality,  the  benefits  expected  from  the  IMIS  will  not  be  realized.  In  other  words,  "garbage  in, 
garbage  out."  There  seems  to  be  a  commonly  held  misconception  that  all  that  is  needed  to  gain 
the  benefits  of  the  IMIS  is  to  convert  the  current  technical  data  into  the  IMIS  format  and  present 
it  on  the  IMIS.  The  problem  is  not  that  simple.  Much  of  the  technical  data  in  use  today  does  not 
have  the  quality  diagnostic  information  required  to  take  full  benefit  of  the  IMIS  diagnostic 
capability.  Additional  information,  and  a  lot  of  "tweaking,"  is  necessary  to  produce  a  quality 
IMIS  data  base.  A  small  effort  to  enhance  the  technical  data  during  the  conversion  process  can 
pay  large  dividends. 

IMIS  and  Maintenance  Force  Structure 

One  major  finding  of  this  study  is  the  demonstration  that  APG  technicians  can  effectively 
perform  many  of  the  tasks  now  limited  to  specialists,  especially  if  they  are  using  an  IMIS.  In  the 
demonstration,  a  relatively  inexperienced  group  of  APG  technicians  (most  at  Skill  Level  3)  using 
the  IMIS  performed  the  troubleshooting  problems  faster  and  used  fewer  spare  parts  than  a 
relatively  experienced  group  of  avionics  specialists  (most  at  Skill  Level  5)  using  the  paper  TO. 

It  is  true  that  the  APG  technicians  required  some  help  to  perform  the  tasks.  However,  these 
helps  were  primarily  basic  system  knowledge  (e.g.,  component  location)  given  to  compensate  for 
the  fact  that  they  had  had  no  training  on  the  test-bed  subsystem.  These  helps  were  given 
primarily  for  the  first  few  problems.  Very  few  helps  were  given  on  their  last  few  problems.  The 
types  of  information  could  easily  be  provided  by  adding  an  orientation  to  the  avionics  systems  to 
the  APG  technical  school.  With  the  IMIS  and  the  addition  of  a  few  days  to  the  APG  technicians 
technical  school,  APG  technicians  could  perform  many  of  the  routine  troubleshooting  tasks  on 
avionics  systems,  freeing  the  avionics  specialists  to  deal  with  the  most  difficult  problems.  This 
approach  would  make  a  reduction  in  the  number  of  specialists  possible  and  would  result  in  major 
savings  in  training  specialists. 

The  demonstrated  ability  of  the  APG  technicians  to  perform  complex  troubleshooting 
tasks  on  avionics  systems  when  using  the  IMIS  provides  strong  support  for  the  Air  Force 
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initiatives  to  reduce  the  number  of  specialties  required  to  maintain  a  weapon  system.  The  idea  is 
to  reduce  specialties  by  broadening  the  areas  of  responsibility  of  the  technicians  to  include  more 
aircraft  subsystems.  The  findings  from  this  study  suggest  that  an  even  greater  reduction  of 
specialties  than  planned  may  be  possible  for  IMIS-supported  weapon  systems.  The 
demonstrated  advantage  of  the  IMIS  for  supporting  the  generalist  technician  makes  the 
implementation  of  an  operational  IMIS  even  more  essential. 

CONCLUSIONS 

The  findings  of  this  study  support  the  following  conclusions  for  F-16  avionics 
subsystems. 

1 .  Fewer  good  parts  are  replaced  when  technicians  use  IMIS  to  perform  fault 
isolation  problems  than  when  they  use  the  TO. 

2.  Technicians  require  less  time  to  complete  fault  isolation  problems  when  they  use 
the  IMIS  than  when  they  use  the  paper  TO. 

3.  Technicians  require  less  time  to  complete  work  order  close-out  and 
documentation  tasks  when  they  use  the  IMIS  than  when  they  use  the  current 
procedures. 

4.  Technicians  make  fewer  serious  errors  when  using  the  IMIS  than  when  using  the 
paper  TO. 

5.  APG  technicians  using  the  IMIS  are  able  to  perform  fault  isolation  problems  as 
effectively  as  specialists  using  the  EMIS  and  more  effectively  than  specialists  using 
the  TO. 

6.  The  effectiveness  of  the  IMIS  for  reducing  part  consumption  varies  from 
subsystem  to  subsystem.  Use  of  the  IMIS  will  result  in  large  savings  in  parts  (and 
dollars)  for  some  subsystems  but  not  for  others.  Additional  research  is  needed  to 
develop  rules  to  determine  which  subsystems  are  likely  to  benefit  most  from  the 
IMIS  or  to  enhance  the  IMIS  diagnostic  module  or  technical  data  to  achieve  the 
same  benefits  for  all  subsystems. 

The  IMIS  field  test  and  demonstration  mark  the  end  of  the  very  successful  IMIS 
demonstration  program.  The  program  developed  and  demonstrated  technology  advances  which 
promise  to  revolutionize  Air  Force  maintenance  operations.  The  concept  and  technology  are 
initially  being  applied  in  the  Joint  Surveillance  Targeting  Attack  Radar  (IS  TARS)  and  F-22  and 
will  be  incorporated  in  the  Integrated  Maintenance  Data  System  (IMDS),  currently  under 
development  for  Air  Force-wide  implementation. 
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ACRONYMS  AND  ABBREVIATIONS 


ADI 

AFTO 

AIP 

APG 

BIT 

CAMS 

CEMS 

COSO 

CND 

DB 

DMT 

EED 

EU 

FI 

FS 

FCR 

FOD 

GAC 

GDES 

HUD 

ICAM 

IDEF 

lETM 

IMIS 

IMIS  DM 

BVUSDS 

IMDS 

INS 

INU 

IPB 

IPS 

JCALS 

JSTARS 

LAN 

MANOVA 

MF'L 

MIW 

MTTR 

MLPRF 

MUX 

NASA  TLX 

PC 

PDU 


Attitude  Director  Indicator 
Air  Force  Technical  Order 
Aircraft  Interface  Panel 
Airplane  General 
Built-in-Test 

Core  Automated  Maintenance  System 
Centralized  Engine  Maintenance  System 
Combat  Oriented  Support  Organization 
Cannot  Duplicate 
Database 

Dual  Mode  Transmitter 
End-to-end  Demonstration 
Electronic  Unit 
Fault  Isolation 
Fighter  Squadron 
Fire  Control  Radar 
Foreign  Object  Damage 
General  Aircraft  Computer 
GDE  Systems 
Heads-up  Display 

Integrated  Computer-Aided  Manufacturing 
ICAM  Definition 

Interactive  Electronic  Technical  Manual 

Integrated  Maintenance  Information  System 

IMIS  Diagnostic  Module 

IMIS  Demonstration  system 

Integrated  Maintenance  Data  System 

Inertial  Navigation  System 

Inertial  Navigation  Unit 

Illustrated  Parts  Breakdown 

lETM  Presentation  System 

Joint  Continuous  Acquisition  Lifecycle  Support 

Joint  Strategic  Targeting  Attack  Radar  System 

Local  Area  Network 

Multiple  Analysis  of  Variance 

Maintenance  Fault  List 

Maintenance  Information  Workstation 

Mean  Time  to  Repair 

Modular  Low  Power  Radio  Frequency 

Multiplex 

National  Aeronautics  and  Space  Agency  Task  Load  Index 

Personal  computer  operating  Microsoft  Disk  Operating  System  (MS  DOS) 

Pilots  Display  Unit 
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PMA 

Portable  Maintenance  Aid 

PSP 

Programmable  Signal  Processor 

RDR 

Radar 

RF 

Radio  Frequency 

RTOK 

Retest  OK 

SBSS 

Standard  Base  Supply  System 

SSS 

System  Segment  Specification 

TO 

Technical  Order 

WD 

Wiring  Diagram 
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APPENDIX  A 


ORDER  OF  PRESENTATION  OF  TEST  PROBLEMS 


Table  A1  presents  the  test  administration  schedule  used  for  data  collection.  The  schedule 
illustrates  the  counterbalancing  of  treatments  (IMIS  vs.  paper  TO)  and  the  randomization  of 
problem  order  used  to  counterbalance  the  order  of  presentation  effects. 
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Table  A1 .  Test  Administration  Schedule  with  Order  of  Presentation  of  Problem  by 

Media  Used 


Subject  / 

Subject  / 

Subject  / 

Subject  / 

Subject  / 

Subject/ 

Media/ 

Media/ 

Media/ 

Media/ 

Media/ 

Media/ 

Problems 

Problems 

Problems 

Problems 

Problems 

Problems 

Week  1- 

Dav  Shift 

Week  2 

Dav  Shift 

Week  3  Dav  Shift 

SI 

S13 

S3 

S15 

S5 

S17 

IMIS 

Paper 

IMIS 

Paper  TO 

Paper  TO 

EMIS 

19 

19 

H7 

H7 

FI 

FI 

FI 

FI 

F3 

F3 

H5 

H5 

H5 

H5 

Ill 

III 

19 

19 

F3 

F3 

H6 

H6 

Ill 

Ill 

Ill 

Ill 

no 

no 

H7 

H7 

H7 

H7 

F2 

F2 

F3 

F3 

Paper  TO 

IMIS 

Paper  TO 

IMIS 

IMIS 

Paper  TO 

H6 

H6 

FI 

FI 

no 

no 

F2 

F2 

H5 

H5 

F4 

F4 

no 

no 

19 

19 

H6 

H6 

H8 

H8 

112 

112 

F2 

F2 

112 

112 

H8 

H8 

112 

112 

F4 

F4 

F4 

H4 

H8 

H8 

Week  1  - 

Niaht  Shift 

Week  2- 

Nisht  Shift 

Week  3  - 

Niaht  Shift 

S2 

S14 

S4 

S16 

S6 

S18 

Paper  TO 

EMIS 

Paper  TO 

EMIS 

EMIS 

Paper  TO 

F2 

F2 

H8 

H8 

no 

no 

110 

no 

112 

112 

H6 

H6 

H6 

H6 

F4 

F4 

F2 

F2 

H7 

H7 

FI 

FI 

Ill 

Ill 

F3 

F3 

H5 

H5 

F3 

F3 

Ill 

Ill 

19 

19 

H7 

H7 

IMIS 

Paper  TO 

IMIS 

Paper  TO 

Paper  TO 

IMIS 

H8 

H8 

Ill 

Ill 

F4 

F4 

112 

112 

H6 

H6 

112 

112 

F4 

F4 

F2 

F2 

H8 

H8 

FI 

FI 

no 

no 

H5 

H5 

H5 

H5 

F3 

F3 

FI 

FI 

19 

19 

H7 

H7 

19 

19 
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Table  Al.  (Continued] 


Subject  / 
Media/ 

Problems 

Subject  / 
Media/ 

Problems 

Subject  / 
Media/ 
Problems 

Subject  / 
Media/ 
Problems 

Subject  / 
Media/ 
Problems 

Subject/ 

Media/ 

Problems 

Week  4- 

•  Dav  Shift 

Weeks 

-  Dav  Shift 

Week  6  Dav  Shift 

S7 

S19 

S9 

S21 

Sll 

S23 

Paper  TO 

IMIS 

Paper  TO 

IMIS 

Paper  TO 

IMIS 

H7 

H7 

FI 

FI 

Ill 

Ill 

Ill 

Ill 

H5 

H5 

H7 

H7 

F3 

F3 

19 

19 

F3 

F3 

H6 

H6 

F3 

F3 

no 

no 

F2 

F2 

112 

112 

F2 

F2 

no 

no 

H7 

H7 

H6 

H6 

IMIS 

Paper  TO 

IMIS 

Paper  TO 

IMIS 

Paper  TO 

FI 

FI 

no 

FI 

H5 

H5 

H5 

H5 

H6 

H5 

19 

19 

19 

19 

F2 

19 

FI 

FI 

F4 

F4 

Ill 

112 

H8 

H8 

112 

112 

F4 

H8 

F4 

F4 

H8 

H8 

H8 

H4 

112 

112 

Week  4  - 

Niaht  Shift 

Week  5  - 

Niaht  Shift 

Week  6  - 

Niaht  Shift 

S8 

S20 

SIO 

S22 

S12 

S24 

IMIS 

Paper  TO 

IMIS 

Paper  TO 

IMIS 

Paper  TO 

H8 

H8 

F2 

F2 

112 

112 

F4 

F4 

no 

no 

F4 

F4 

112 

112 

H6 

H6 

H8 

H8 

FI 

FI 

Ill 

Ill 

H5 

H5 

H5 

H5 

H7 

H7 

19 

19 

19 

19 

F3 

F3 

FI 

FI 

IMIS 

IMIS 

Paper  TO 

IMIS 

Paper  TO 

EVflS 

F2 

F2 

112 

112 

H6 

H6 

Ill 

Ill 

F4 

F4 

F2 

F2 

H6 

H6 

H8 

H8 

Ill 

Ill 

no 

no 

H5 

H5 

F3 

F3 

H7 

H7 

19 

19 

H7 

H7 

F3 

F3 

FI 

FI 

no 

no 

82 


r 

APPENDIX  B 

.  SAMPLE  DATA  COLLECTION  FORMS 
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IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
HUD  AE  (MFL  002)  /  ELECTRONIC 

PROBLEM#  H6-E  SUBJECT# .  DATE: .  D/C: .  T/O:  JJ/BH 

LOG  IN  USING  EMP#  01201  (DAVIS) 

INS  Present  Position  =  Latitude:  N  33221  Longitude:  W  112224  Altitude:  1080 


EVENT 


TIME 


COMMENTS 


ADJ 


Task  Assignment 

Arrive  at  A/C 

Questions  to  "Pilot"?  Y  /  N 

Fault  Verification  start 

PINOUT 

BIT  (AUTO)  start 

Access:  Connect  1553? 

(Outcome:  MFL  HUD002) 

finish 

Y/N 

finish 

Test  start 

MUX  ckt  =  2.0  ohms 

PIN  IN 

(Outcome:  OK) 

finish 

Rect  (HUDEU)  start 

Order  part  start 

Pro-Super  Co-ord 

finish 

Remove  start 

finish 

Install  start 

finish 

finish 

System  Health  start 

HUD  BIT  (AUTO)  start 

♦SIMULATED*  finish 

HUD  OPS  Ck  start 

♦SIMULATED*  finish 

finish 
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IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
HUD  AE  (MFL  002)  /  ELECTRONIC 

PROBLEM#  H6-E  SUBJECT# .  DATE: .  D/C: .  T/0:  JJ/BH 


LOG  IN  USING  EMP#  01201  (DAVIS) 

INS  Present  Position  =  Latitude:  N  33221  Longitude:  W  112224  Altitude:  1080 


Job  Closeout  start 

(Follow-ons  not  req'd) 

finish 

Task  Complete 

Closeout  MSG  Rec'd? 

Y/N 

CLOSEOUT  MASTER  INFO: 

TM:  B 
WUC:  74BU0 
AT:  R 

HOW  MAL:  644,  290,  242,  255,  or  others  applicable  in  -06  manual 
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IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
HUD  AE  (MFL  002)  /  PAPER 


PROBLEM#  H6-P  SUBJECT# . DATE:. 


D/C: . .  T/0:  JJ/BH 


INS  Present  Position  =  Latitude:  N  33221 

EVENT  TIME 


Task  Assignment 


Check  out  TOs 


Arrive  at  A/C 


Questions  to  "Pilot"?  Y  /  N 


Fault  Verification  start 


HUD  OPS  Ck  start  _ 

(JG94-72-01) 

Access:  Power  On?  Y  /  N 

WPN  Sys  Initialization?  Y  /  N 

WPN  Sys  Fault  Detection?  Y  /  N 

INS  Aligmnent?  Y  /  N 

(Outcome;  MFL  HUD002) 

finish 


finish 


Testl  (n94-72/T-l)  start  _ 

Assoc  MUX  faults? 


(Outcome:  NO) 

finish 


Test  2  (FI94-72/T-2)  start 
MUX  ckt  =  2.0  ohms 


(Outcome:  OK) 


Rect(HUDEU) 

(JG94-72-06) 


Order  part 

start 

(9472A1) 

finish 

Remove 

start 

finish 

Install 

start 

finish 

finish 

Longitude:  W  112224  Altitude:  1080 

COMMENTS  ADJ 


PINOUT 


Add'l  TO  research  req'd?  N  /  Y;  time.. 


PINOUT 


Add'l  TO  research  req'd?  N  /  Y;  time.. 


PIN  IN 


Add'l  TO  research  req'd?  N  /  Y;  time.. 


Add'l  TO  research  req'd?  N  /  Y;  time. 


IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
HUD  AE  (MFL  002)  /  PAPER 


PROBLEM#  H6-P  SUBJECT# .  DATE: .  D/C: .  T/0:  JJ/BH 


INS  Present  Position  =  Latitude:  N  33221  Longitude:  W  112224  Altitude:  1080 


System  Health  start 

PIN  IN 

Add'l  TO  research  req’d?  N  /  Y;  time . 

HUD  BIT  start 

(JG94-72-01) 

♦SIMULATED*  finish 

HUD  OPS  ck  start 

(JG94-72-01) 

♦SIMULATED*  finish 

finish 

Job  Closeout  start 

finish 

Task  Complete 

CLOSEOUT  MASTER  INFO: 

TM:  B 
WUC:  74BU0 
AT:  R 

HOW  MAL:  644,  290,  242,  255,  or  others  applicable  in  -06  manual 
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IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
INS  AG:  145/100  (MFL  Oil)  /  ELECTRONIC 

PROBLEM#  II 1-E  SUBJECT# .  DATE: .  D/C: .  T/0:  JJ/BH 

LOG  IN  USING  EMP#  00998  (GREATHOUSE) 

INS  Present  Position  =  Latitude:  N  33221  Longitude:  W  112224  Altitude:  1080 


EVENT 


TIME 


COMMENTS 


ADJ 


Task  Assignment 

Arrive  at  A/C 

Questions  to  "Pilot"? 

Y/N 

Fault  Verification 

start 

SHORT  IN 

BIT  (AUTO) 

start 

Access:  Connect  1553? 

Y/N 

(Outcome:  MFL  INSOll) 

finish 

finish 

Test  1 

start 

SHORT  IN 

INSM  144/145  (AUTO) 

(Outcome:  145/100) 

finish 

Rect  1  (INU) 

start 

Order  part 

start 

Pro-Super  Co-ord 

finish 

Remove 

start 

finish 

Install 

start 

finish 

finish 

System  Health 

Start 

SHORT  IN 

INS  Alignment 

start 

Access:  Power  On? 

Y/N 

WPN  Sys  Initialization? 

Y/N 

WPN  Sys  Fault  Detection? 

Y/N 

(Outcome:  FAIL;  INSOll) 

finish 

finish 
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IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
INS  AG:  145/100  (MFL  Oil)  /  ELECTRONIC 


PROBLEM#  II 1-E  SUBJECT# .  DATE: .  D/C: .  T/0:  JJ/BH 

LOG  IN  USING  EMP#  00998  (GREATHOUSE) 

INS  Present  Position  =  Latitude:  N  33221  Longitude:  W  112224  Altitude:  1080 


Test  2  start 

INU  ->  HSI  continuity 
♦SIMULATED* 


SHORT  OUT 


(OUTCOME:  OK) 


finish 

Rect  2  (HSI)  start 

Order  part  start 

Pro-Super  Co-ord 

_ finish 

Remove  start 

finish 

Install  start 

finish 

_ finish _ 

System  Health _ start _  SHORT  OUT 

INS  Alignment  start 

♦SIMULATED*  finish 

MAG  HDG  /  Steering  start 
♦SIMULATED*  finish 

INS  Stored  HDG  start 

♦SIMULATED*  finish 

_ finish 

Job  Closeout  start 


(Follow-ons  not  req'd) 

_ finish _ 

Task  Complete 

Closeout  MSG  Rec'd?  Y  /  N 


CLOSEOUT  MASTER  INFO: 


TM:  B 
WUC:  51BA0 
AT:  R 

HOW  MAL:  290,  657,  658,  242,  255,  or  others  applicable  in  -06  manual 
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IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
INS  AG:  145/100  (MFL  Oil)  /  PAPER 


PROBLEM#  II 1-P  SUBJECT# .  DATE .  D/C:  T/0:  JJ/BH 

INS  Present  Position  =  Latitude;  N  33221  Longitude;  W  112224  Altitude:  1080 


EVENT 


Task  Assignment 


Check  out  TOs 


Arrive  at  A/C 


Questions  to  "Pilot"? 


Fault  Verification  start 


INS  Alignment  start 

(JG94-63-01) 

Access;  Power  On? 

WPN  Sys  Initialization? 
WPN  Sys  Fault  Detection? 

(Outcome:  MFL  INSOll) 

finish 


finish 


Testl  (FI94-63/T-1)  start 
INS  Alignment 

Access:  Power  On? 

WPN  Sys  Initialization? 
WPN  Sys  Fault  Detection? 

(Outcome:  FAIL;  INSOl  1) 

finish 


Rect  1  (INU)  start 

(JG94-63-03) 


TIME 


Order  part 

start 

(9463A1) 

finish 

Remove 

start 

finish 

Install 

start 

finish 

finish 

COMMENTS 


SHORT  IN 

Add'l  TO  research  req'd? 

N/Y;  time . 

SHORT  IN 

Add'l  TO  research  req'd? 

N/Y;time . 

Add'l  TO  research  req'd? 

N/Y;  time . 
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IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
INS  AG:  145/100  (MFL  Oil)  /  PAPER 


PROBLEM#  II 1-P  SUBJECT# .  DATE .  D/C: .  T/0:  JJ/BH 

INS  Present  Position  =  Latitude:  N  33221  Longitude:  W  112224  Altitude:  1080 
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IMIS  CONSTRAINED  TEST  DATA  COLLECTION  SHEET 
INS  AG:  145/100  (MFL  Oil)  /  PAPER 

PROBLEM#  II 1-P  SUBJECT# .  DATE .  D/C: .  T/0:  JJ/BH 

INS  Present  Position  =  Latitude:  N  33221  Longitude:  W  112224  Altitude:  1080 


System  Health _ start 

INS  Alignment  start 

(JG94-63-01) 

♦SIMULATED*  finish 

MAG  HDG  /  Steering  start 
(JG94-63-01) 

♦SIMULATED*  finish 

INS  Stored  HDG  start 
(JG94-63-01) 

♦SIMULATED*  finish 

_ finish 

Job  Closeout  start 


SHORT  OUT 


Add'l  TO  research  req'd?  N  /  Y;  time. 


finish 

Task  Complete _ 


CLOSEOUT  MASTER  INFO: 

TM:  B 
WUC:  51BA0 
AT:  R 
HOWMAL: 


290,  657,  658,  242,  255,  or  others  applicable  in  -06  manual 


APPEDIX  C 


MEANS,  ANALYSIS  OF  VARIANCE,  AND  SHEFFE  PAIRED  COMPARISON  TABLES 

Variable:  Percent  of  Problems  Successfully  Completed 

Tabled.  Mean  Percent  of  Problems 
Completed 


TO 

IMIS 

Valid  N 

Specialist 

81.94 

100 

72 

APG 

69.44 

98.61 

72 

All  Groups 

75.69 

99.31 

144 

Table  C2.  Analysis  of  Variance  for  Percent  of  Problems  Successfully  Completed 


Effect 

df 

Effect 

MS 

Effect 

df 

Error 

MS 

Error 

F 

p-level 

Specialty 

1 

0.3472 

142 

0.0997 

3.4838 

0.0640 

Media 

1 

4.0139 

142 

0.0899 

44.6551 

0.0000 

Spec.  X  Media 

1 

0.2222 

142 

0.0899 

2.4723 

0.1181 

Table  C3.  Probability  Matrix  for  Sheflfe  Post-Hoc  Test  of  Differences  Between 
Means  for  Percent  of  Problems  Completed 


Means 

{1} 

0.8194 

{2} 

1.0000 

{3} 

0.6944 

{4} 

0.9861 

Specialist  TO 

{1) 

0.0058 

0.1047 

0.0132 

Specialist  IMIS 

{2} 

0.0058 

0.0000 

0.9944 

APG/TO 

{3) 

0.1047 

0.0000 

0.0000 

APG/IMIS 

(4} 

0.0132 

0.9944 

0.0000 

Variable:  Number  of  Parts  Used 


Table  C4.  Mean  Number  of  Parts  Used  by 
_ Each  Subject _ 


TO 

IMIS 

Valid  N 

Specialist 

8.67 

6.42 

12 

APG  Technician 

8.30 

5.30 

10 

All  Groups 

8.50 

5.91 

22 

Note;  This  analysis  was  based  upon  the  total  number  of  parts  used  by  each  technician.  Thus,  the 
maximum  N  was  12  per  subject  category.  Data  for  two  APG  technicians  were  not  used  due  to 
incomplete  data  from  failing  to  satisfactorily  complete  a  problem. 


Table  C5.  Analysis  of  Variance  for  Percent  of  Problems  Successfully  Completed 


Effect 

df 

Effect 

MS 

Effect 

df 

Error 

MS 

Error 

F 

p-level 

Specialty 

1 

6.0008 

20 

1.5329 

3.9146 

0.0618 

Media 

1 

75.1705 

20 

1.4563 

51.6192 

0.0000 

Spec.  X  Media 

1 

1.5341 

20 

1.4563 

1.0535 

0.3170 

Table  C6.  Probability  Matrix  for  Sheffe  Post-Hoc  Test  of  Differences  Between 


Means  for  Percent  of  Problems  Successfully  Completed 


Means 

{1} 

8.6667 

{2} 

6.4167 

{3} 

8.3000 

{4} 

5.3000 

Specialist  TO 

(1) 

0.0022 

Specialist  IMIS 

{2} 

0.0022 

APG/TO 

(3) 

0.9168 

0.0153 

0.0003 

APG/IMIS 

(4} 

0.0000 

0.2310 

0.0003 

Variable:  Serious  Errors 

Table  C7.  Mean  Number  of  Serious  Errors 


Per  Problem 


TO 

IMIS 

MAJOR 

Specialist 

0.69 

0.29 

70 

APG 

1.06 

0.18 

72 

All  Groups 

0.87 

0.23 

142 

94 


Table  C8.  Analysis  of  Variance  for  Mean  Number  of  Serious  Errors  Per  Problem 


Effect 

df 

Effect 

MS 

Effect 

df 

Error 

MS 

Error 

F 

p-level 

Specialty 

1 

1.2433 

140 

0.9033 

1.3763 

0.2427 

Media 

1 

28.8491 

140 

0.8524 

33.8442 

0.0000 

Spec.  X  Media 

1 

4.0041 

140 

0.8524 

4.6973 

0.0319 

Table  C9.  Probability  Matrix  for  Shefife  Post-Hoc  Test  of  Differences  Between 
_ Means  for  Serious  Errors  per  Problem _ 


Means 

{1} 

0.6857 

{2} 

0.2857 

{3} 

1.0556 

{4} 

0.1806 

Specialist  TO 

{1) 

0.0919 

0.1326 

Specialist  IMIS 

{2} 

0.0919 

0.0000 

mmm 

APG/TO 

(3) 

0.1326 

0.0000 

0.0000 

APG/IMIS 

{4} 

0.0163 

0.9273 

0.0000 

Variable:  Time  to  Complete 


TO 

IMIS 

Valid  N 

Specialist 

149.29 

123.64 

59 

APG 

175.82 

124.04 

50 

All  Groups 

161.46 

123.83 
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Table  Cl  1. 

Analysis  of  Variance  for  Time  to  Complete 

df 

MS 

df 

MS 

Effect 

Effect 

Effect 

Error 

Error 

F 

p-level 

Specialty 

1 

9,812.22 

107 

2,517.83 

3.8971 

0.0509 

Media 

1 

81,118.05 

107 

1,287.15 

63.0214 

0.0000 

Spec.  X  Media  1 

9,243.61 

107 

1,287.15 

7.1815 

0.0085 

Table  C12.  Probability  Matrix  for  Sheffe  Post-Hoc  Test  of  Differences  Between 


Means  for  Time  to  Complete 

Means 

{1} 

149.2881 

{2} 

123.6441 

{3} 

175.8200 

{4} 

124.0400 

Specialist  TO 

(1) 

0.0027 

0.0030 

0.0054 

Specialist  IMIS 

{2} 

0.0027 

0.0000 

0.9999 

APG/TO 

{3) 

0.0030 

0.0000 

0.0000 

APG/IMIS 

(4) 

0.0054 

0.9999 

0.0000 

Variable:  Preparation  Time 


Table  C 1 3 .  Means  for  Preparation  Time 

_ TO  IMIS  Valid  N 

Specialist  5  5.1806  72 

APG  5.0972  4.5556  72 


All  Groups 


5.0486  4.8681 


Table  Cl  4.  Analysis  of  Variance  for  Preparation  Time 


Effect 


Specialty 

Media 

Spec.  X  Media 


df 

Effect 


1 

1 

1 


MS 

Effect 


5.01 

2.35 

9.39 


df 
Error 


142 

142 

142 


0.7570 

0.4175 

1.6702 


0.3857 

0.5192 

0.1983 


Table  Cl  5.  Probability  Matrix  for  Sheffe  Post-Hoc  Test  of  Differences  Between 

Means  for  Preparation  Time 


Variable:  Fault  Verification  Time 

Table  C16.  Means  for  Fault  Verification  Time 


TO 

IMIS 

Valid  N 

Specialist 

9.65 

10.57 

72 

APG 

18.31 

11.07 

72 

All  Groups 

13.98 

10.82 

144 

Table  C17.  Analysis  of  Variance  for  Fault  Verification  Time 


Effect 


Specialty 

Media 

Spec.  X  Media 


df 

Effect 


1 

1 

1 


MS 

Effect 


1,507.92 

718.84 

1,196.42 


df 

Error 


142 

142 

142 


49,15  30.6784  0.0000 

41.59  17.2825  0.0001 

41.59  28.7648  0.0000 
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Table  C18.  Probability  Matrix  for  Sheffe  Post-Hoc  Test  of  Differences  Between 

Means  for  Fault  Verification  Time 


Means 

{1} 

9.6528 

{2} 

10.5694 

{3} 

18.3056 

{4} 

11.0694 

Specialist  TO 

{1) 

0.8666 

0.0000 

0.6297 

Specialist  IMIS 

{2} 

0.8666 

0.0000 

0.9748 

APG/TO 

{3) 

0.0000 

0.0000 

0.0000 

APG/IMIS 

(4} 

0.6297 

0.9748 

0.0000 

Variable:  Fault  Isolation  Time 

Table  Cl 9.  Means  for  Fault  Isolation  Time 

TO 

IMIS 

Valid  N 

Specialist  116.22 

95.92 

59 

APG  132.72 

96.86 

50 

All  Groups  123.79 

96.35 

109 

Table  C20.  Analysis  of  Variance  for  Fault  Isolation  Time 

Effect 

df 

Effect 

MS 

Effect 

df 

Error 

MS 

Error 

F 

p-level 

Specialty 

1 

4,117.92 

107 

2,371.72 

1.7363 

0.1904 

Media 

1 

42,687.27 

107 

1,084.93 

39.3457 

0.0000 

Spec.  X  Media 

1 

3,274.17 

107 

1,084.93 

3.0179 

0.0852 

Table  C21 .  Probability  Matrix  for  Sheffe  Post-Hoc  Test  of  Differences  Between 

Means  for  Fault  Isolation  Time 

Means 

{1} 

116.2203 

{2} 

95.9153 

{3} 

132.7200 

{4} 

96.8600 

Specialist  TO 

(1) 

0.0134 

0.0852 

0.0292 

Specialist  IMIS 

(2} 

0.0134 

0.0000 

0.9991 

APG/TO 

(3) 

0.0852 

0.0000 

0.0000 

APG/IMIS 

JIJL 

0.0292 

0.9991 

0.0000 
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Variable:  Part  Ordering  Time 


Table  C22.  Mean  Time  to  Order  Each  Part 


Time 

Valid  N 

Specialist  TO 

19.42 

53 

APG  TO 

25.28 

51 

Specialist  IMIS 

1.16 

45 

APG  IMIS 

1.47 

37 

All  Groups 

13.04 

186 

Table  C23.  Analysis  of  Variance  for  Part  Ordering  Time 


Effect 

df 

Effect 

MS 

Effect 

df 

Error 

MS 

Error 

F 

p-level 

Specialty 

1 

1.24 

140 

0.90 

1.3763 

0.2427 

Media 

1 

28.85 

140 

0.85 

33.8442 

0.0000 

Spec.  X  Media 

1 

4.00 

140 

0.85 

4.6973 

0.0319 

Table  C24.  Probability  Matrix  for  ShefFe  Post-Hoc  Test  of  Differences  Between 


Means  for  Part  Ordering  Time 


Means 

{1} 

19.4198 

{2} 

25.2827 

{3} 

1.1611 

{4} 

1.4730 

Specialist  TO 

{1) 

0.0000 

0.0000 

0.0000 

Specialist  IMIS 

{2} 

0.0000 

0.0000 

0.0000 

APG/TO 

{3) 

0.0000 

0.0000 

0.9788 

APG/IMIS 

J4L_ 

0.0000 

0.0000 

0.9788 

Variable:  Close-Out  Time 


Table  C25.  Means  Close-Out  Time 


TO 

IMIS 

Valid  N 

Specialist 

14.667 

8.1667 

72 

APG 

17.31 

8.8169 

71 

All  Groups 

15.979 

8.4895 

143 

98 


Table  C26.  Analysis  of  Variance  for  Close-Out  Time 


Effect 

df 

Effect 

MS 

Effect 

df 

Error 

MS 

Error 

F 

p-level 

Specialty 

1 

193.88 

141 

9.36 

20.7104 

0.0000 

Media 

1 

4,017.90 

141 

14.11 

284.7037 

0.0000 

Spec.  X  Media 

1 

70.99 

141 

14.11 

5.0305 

0.0265 

Table  C27.  Probability  Matrix  for  Sheflfe  Post-Hoc  Test  of  Differences  Between 

Means  for  Close-Out  Time 

Means 

{1} 

14.6667 

{2} 

8.1667 

{3} 

17.3099 

{4} 

8.8169 

Specialist  TO 

(1) 

0.0000 

0.0008 

0.0000 

Specialist  IMIS 

{2} 

0.0000 

0.0000 

0.7841 

APG/TO 

(3) 

0.0008 

0.0000 

0.0000 

APG/IMIS 

J4} 

0.0000 

0.7841 

0.0000 

Variable:  Documentation  Errors 

Table  C28.  Mean  Number  of  Documentation 


Errors  Per  Problem 


TO 

IMIS 

Valid  N 

Specialist 

0.0143 

0.1143 

70 

APG 

0.0833 

0.0417 

72 

All  Groups 

0.0493 

0.0775 

142 

Table  C29.  Analysis  of  Variance  for  Documentation  Errors 


Effect 

df 

Effect 

MS 

Effect 

df 

Error 

MS 

Error 

F 

p-level 

Specialty 

1 

0.0002 

140 

0.0776 

0.0029 

0.9570 

Media 

1 

0.0604 

140 

0.0828 

0.7296 

0.3945 

Spec.  X  Media 

1 

0.3562 

140 

0.0828 

4.3031 

0.0399 

Table  C30.  Probability  Matrix  for  Sheffe  Post-Hoc  Test  of  Differences  Between 

Means  for  Documentation  Errors 

{1} 

{2} 

{3} 

{4} 

Means 

0.0143 

0.1143 

0.0833 

0.0417 

Specialist  TO 

{1) 

0.2426 

0.5648 

0.9558 

Specialist  IMIS 

{2} 

0.2426 

0.9378 

0.5219 

APG/TO 

{3) 

0.5648 

0.9378 

0.8600 

APG/IMIS 

(4) 

0.9558 

0.5219 

0.8600 

99 


APPENDIX  D 


EXIT  QUESTIONNAIRE  COMMENTS 
Exit  Questionnaire  Comments 

Question:  What  did  you  like  about  IMIS  as  an  aid  to  doing  your  job? 

The  speed  with  which  you  and  IMIS  can  do  a  job,  the  graphics  were  good  for  aiding  with 
wire  troubleshooting. 

The  whole  concept  is  pretty  good.  Go  to  aircraft,  diagnose,  order  part,  not  have  to  carry 
paper. 

The  speed  in  diagnosing  and  accomplishing  a  task. 

When  the  PMA  was  connected  to  the  1 553  Bus,  it  was  very  quick  and  accurate. 

Parts  ordering,  debriefing  and  job  close-out  were  much  simpler  than  using  CAMS. 

Information  easily  accessible,  accurate,  easy  to  understand. 

It  cut  down  on  the  mental  fiaistration  of  my  job  and  the  amount  of  tech  data  that  I  would 
normally  have  had  to  go  through  to  do  my  job.  Also,  the  part  ordering  feature  cut 
down  part  ordering  time  considerably. 

It  cut  the  time  down  to  nothing. 

You  had  one  unit  instead  of  many  TOs.  It  gives  you  only  necessary  information,  less 
confusion. 

It  makes  sure  everything  is  done  in  order. 

The  integration  of  all  TOs  and  the  ease  of  ordering  parts. 

The  troubleshooting  fault  tree  and  the  job  guides.  They  replicated  the  paper  ones  very 
well. 
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It  was  very  straight  forward.  It  took  everything  associated  with  the  system  into 

consideration,  and  gave  the  choice  to  the  technician  as  to  the  action  he  feels  to  be 
best.  It  made  ordering  parts  very  simple,  actually  it  does  it  for  you.  It  saves  time  on 
paper  work.  I  feel  with  DVUS  that  anybody,  regardless  of  experience,  can  work  on 
the  aircraft. 

The  ease  of  closing  work  orders  and  the  ordering  of  parts.  The  troubleshooting  was  also 
helpful. 

Simplified  troubleshooting. 

Easier  than  carrying  a  library  of  TOs. 

Saved  effort  in  fumbling  through  books  when  shooting  wires. 

Convenient  way  to  order  parts. 

Integration  with  aircraft  systems  (BIT  checks),  faster  than  the  aircraft  (GAC). 

Made  troubleshooting  easier. 

It  made  troubleshooting  much  easier  and  eliminated  tedious  job  of  checking  out  tech  data 
and  equipment  from  support.  I  also  feel  that  IMIS  will  decrease  the  down  time  of  the 
aircraft. 

IMIS  was  very  helpful  in  all  aspects  of  completing  a  job  from  starting  the  work  order,  to 
verifying  a  fault,  and  to  rectifying  the  fault.  The  most  impressive  aspect  of  IMIS  is 
parts  order! 

If  the  system  is  used  on  the  flightline,  it  could  speed  up  different  jobs  and  take  away  a  lot  of 
other  fiustration  with  dealing  with  the  different  TOs. 

The  auto  BIT  feature  was  extremely  helpful,  and  the  recommended  best  action  was  a  good 
feature  as  well. 

It  made  unknown  tasks  easy  to  understand;  parts  ordering  an  job  closings  were  fast. 

I  found  it  very  helpful.  With  me  being  fairly  new  to  my  job,  I  hadn't  used  hardly  any  of  the 
TO  material,  as  far  as  FI  are  concerned.  From  just  learning  to  use  both,  after  some 
basic  TO  (paper)  training  I  found  IMIS  much  more  easier. 

It  takes  up  the  repetitious  workload. 

Parts  ordering,  CAMS  interface,  not  having  to  carry  a  lot  of  books. 
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Makes  troubleshooting  very  easy. 
It  works  very  well  as  aid. 


Question:  What  did  you  dislike  about  IMIS. 

There  is  nothing  I  can  think  of  that  would  make  me  dislike  this  system. 

Transition  times  between  screens. 

Some  text  is  misleading.  For  example,  hook  up  of  the  PMA  to  A/C  could  be  clearer.  INS 
task  seemed  to  require  more  thought. 

I  personally  read  over  the  job  guide  before  I  do  a  job  With  IMIS,  you  can't  really  do  that. 

Periodic  delays  between  screens. 

It  still  had  lock-ups  and  that  you  did  not  have  the  option  to  go  back  if  you  happened  to  hit 
the  'wrong'  button  during  operational  checks. 

Typical  prototype  bugs. 

The  user  interface.  The  hourglass  should  consistently  go  to  a  place  on  the  screen  where  it 
is  visible.  As,  as  the  PMA  is  transitioning  is  quite  confusing.  There  is  no  cue  thatthe 
machine  is  busy.  It  actually  prints  "Press  FI  for  OK,"  when  you  can't.  Also,  having 
three  methods  to  select  an  option  is  not  a  good  idea.  Use  FI,  F8„  and  Select 
consistently,  not  FI  for  this  F8  for  that  and  Select  for  something  else.  Use  either  the 
thumb  knob  or  arrow  keys. 

The  fact  that  steps  were  given  only  one  step  at  a  time  in  the  job  guide  portion.  It  can  slow 
down  a  job. 

It  was  a  little  heavy.  A  keyboard  would  have  been  convenient.  When  troubleshooting 
wires,  it  could  be  more  specific  about  the  whereabouts  of  some  of  the  components, 
i.e.,  "station  88  disconnect...  "  A  little  slow  at  times. 

The  speed  of  the  software.  No  link  between  part  number  and  serial  number  removed  (with 
work  order  close-out). 

IMIS  is  slow. 

Slow  to  process  information  at  times. 
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Keys  sometimes  hard  to  press;  desired  result  doesn't  occur. 

At  times  was  a  little  slow. 

In  the  short  time  I  used  the  IMIS,  I  can't  really  say  that  I  disliked  anything  about  the 
system. 

The  pros  definitely  outweigh  the  cons.  However,  IMIS  seems  to  sometimes  get  into  slow 
modes,  I.  e.,  changing  from  screen  to  screen,  giving  completed  task  information. 

I  did  not  like  the  reliability  factor.  If  you  are  in  the  middle  of  a  job  and  it  goes  down,  it 
tends  to  be  fhistrating. 

The  time  it  took  to  process  during  RF  operations,  the  amount  of  breakdowns,  and  a  few 
minor  misleading  or  irritating  display  errors. 

The  screen  is  difficult  to  see  unless  you  are  looking  at  just  the  right  angle.  Sometimes 
when  it  is  thinking,  it  doesn't  let  you  know  what  it  is  doing  and  you  are  apt  to  get 
fhistrated  with  it  when  you  try  to  do  something  and  it  wont  respond. 

About  the  only  thing  I  didn't  like  about  IMIS  is  the  thumb  control  for  the  cursor. 

Nothing. 

Speed,  poor  pictures,  one  step  at  a  time.  A  page  down  or  up  t  proceed  would  be  nice. 

It's  too  hard  to  get  back  to  a  step  you  miss.  It  takes  to  long  to  change  screens  at  certain 
times. 

How  slow  it  is  in  processing  information!  Having  to  skip  from  FI  to  the  select  key. 

Question:  What  Changes  would  you  make  to  improve  the  system? 

None. 

Less  redundancy,  e.g.,  safe  for  maintenance  question  being  asked  over  and  over.  Make 
safe  for  maintenance  an  option  to  get  to  if  you  want  to. 

Allow  the  maintainer  the  option  to  look  over  the  job  before  doing  a  job,  so  there  are  no 
surprises. 

Speed  of  processing  improvements. 

Try  and  add  some  more  prompts  to  the  screens  on  choices.  Breakdown  pictures  of  LRUs 
could  be  a  little  more  clear.  Color  screens  would  be  great. 
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More  detailed  pictures,  faster  response  time. 

Fix  user  interface  (see  pre\dous  user  interface  comments). 

Speed  up  the  reaction  time.  It  was  slow  at  times. 

Speed  up  the  PMAs. 

Continuity  in  using  the  system,  i.e.,  no  shifting  between  FI  and  SELECT  keys. 

Give  better  instructions  for  some  of  the  more  obscure  items,  for  example,  disconnects  and 
transformer  assemblies. 

If  you  remove  a  part  it  should  update  the  work  order.  The  speed  and  everything  else  was 
fine. 

Process  faster. 

Add  schematic/wiring  diagram  type  diagrams.  This  is  a  must.  From  time  to  time  we  see  a 
problem  that  "can't  happen"  according  to  the  FI.  In  other  words,  there  is  nothing 
listed  for  "Pilot  Detectable"  or  "Maintenance  Detectable"  faults.  Thus  there  is  no 
fault  tree  to  follow.  Since  the  avionics  systems  of  the  F- 16  are  so  integrated,  a 
component  not  in  the  "suspected  bad"  system  can  cause  a  malfunction.  The  only 
way,  in  most  cases,  to  troubleshoot  these  kind  of  discrepancies  is  to  break  out  the 
schematics  and  "chase"  the  signal  flow. 

Response  time. 

Other  than  a  few  minor  computer  glitches,  the  system  worked  well.  It  would  be  nice  if  the 
PMA  could  operate  longer  on  a  charged  battery. 

Make  the  system  convert  from  page  to  page  faster  and  fix  the  RF  problems. 

Improve  reliability  and  response  time. 

Increase  the  amount  of  back  light,  drop  a  glare  shield  on  it  (maybe  with  solar  panels),  put  a 
zoom  feature  on  the  pictures. 

Try  to  speed  it  up  on  some  things,  make  the  keyboard  a  little  more  user  fnendly,  unless  you 
are  really  carefiil,  the  screws  can  become  a  FOD  hazard  in  the  cockpit. 

I  think  that  maybe  a  separate  detachable  mouse  or  maybe  a  track  ball  would  be  good 
improvement. 
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Quicker  response  times  to  the  commands. 

Speed. 

Use  a  process  or  unit  to  speed  it  up. 

Make  the  system  memory  faster.  Sometimes  it  takes'  awhile  between  choices. 
Speed  up  some  of  the  reaction  times.  It  was  slow  at  some  times. 


Question:  Other  comments? 

Good/excellent  idea. 

IMIS  would  be  an  outstanding  system  to  supplement  existing  tech  data.  However,  its 
ability  to  present  the  "big  picture"  of  a  problem  appears  to  be  limited  and  it  does  not 
explain  why  something  is  happening. 

Thank  you  for  the  training  on  the  jet.  The  hands  on  troubleshooting  was  very  helpful 
People  "in  charge"  were  friendly  and  informative  as  to  what  was  going  on  with  the 
program. 

It  is  difficult  to  form  a  real  opinion  of  the  system  at  this  time,  as  a  fair  judgment  can't  really 
be  made  until  the  system  is  truly  "in  the  field."  But,  I  like  what  I  see  so  far,  and  have 
every  bit  of  confidence  that  your  team  will  perfect  this  system  and  put  it  into  use.  It 
has  been  a  pleasure  to  work  with  you. 

Overall  a  good  system. 

IMIS  definitely  needs  to  be  an  integral  part  of  the  AF.  The  AF  will  benefit  greatly  fi'om  the 
implementation  of  IMIS. 

I  think  this  program  is  a  good  one  and  has  been  very  educational  for  me. 

Great  project  -  quite  a  bit  better  than  having  to  use  TOs.  I  hope  I  see  it  in  use  somewhere, 
sometime  before  I  get  out  of  the  service  or  even  in  the  civilian  world. 

I  really  hope  this  system  takes  effect  soon. 

Very  good  unit.  If  it  were  not  for  the  waiting  on  the  computer,  it  would  be  great. 

I  like  the  idea  and  the  concept!  I  hope  to  see  it  in  the  future  on  all  A/C  systems. 
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